Guest post by the Data Science @ NLM Training Program team.
As part of our effort to advance Goal 3 of the NLM Strategic Plan (“Build a workforce for data driven research and health”), NLM launched the Data Science @ NLM (DS@NLM) Training Program in 2019 to help ensure that all staff are prepared to engage with and participate in NLM’s developing data science efforts.
Our efforts have stayed on track despite the changes caused by the COVID-19 pandemic, and we’re proud to highlight DS@NLM events held during the past year. We’re also sharing lessons learned throughout the training program, which are applicable to any individual or organization trying to help develop data science skills in the fields of health and biomedical information.
Earlier this month, we marked two years of the DS@NLM Training Program with a Spring Fling series of virtual events celebrating the data science training achievements of NLM staff.
Our Spring Fling kicked off with “lightning talk” presentations featuring several graduates of our intensive Data Science Fundamentals course, who shared their final class projects with NLM colleagues. Participants in our year-long Data Science Mentorship program also had the opportunity to present their Capstone projects. Our program mentees, who were mentored by NLM staff members, developed their data science skills by completing projects that applied data science techniques to help improve NLM operations.
What We’ve Learned:
Be responsive to specific needs; one size does NOT fit all.
Data plays a role in virtually everything we do at NLM, and as we aim to provide data training opportunities for staff working in many different areas, we recognize that different staff members have unique training needs. New training opportunities for some staff, such as our researchers, may hinge on their knowledge of machine learning. Metadata specialists may have more need for data cleaning or text processing skills, while administrators may benefit more from learning about data visualization.
People also learn in different ways, be it through shorter webinars and workshops, longer intensive courses, or self-directed learning. The DS@NLM program provides a variety of activities to meet these needs, including opportunities for various skill levels and topics, from short webinars to on-demand classes to ten-week intensive training courses.
Be responsive to staff feedback; give people what they ask for.
To help us determine what to offer, we engaged directly with our audience, asking NLM staff what they needed and listening to their responses. Because of the wide variety of work done at NLM, receiving feedback from staff helped us better understand their specific training needs. While we cannot always offer individualized programs to meet every need, staff feedback always helps us discover new ideas for future programming.
Teaching skills is just the beginning; applying new skills is essential.
A key lesson learned from staff feedback is that teaching new data skills is important, but that’s not enough on its own; teaching how to put newly acquired data skills to use in the real world or applying it to their work is just as important. Helping staff learn to apply data science techniques to their work transforms this new knowledge from theoretical to practical. The Data Science Mentorship Program, with its concluding Capstone project, is a great example of an opportunity for staff to both develop skills and practice applying them.
We applaud and celebrate all the hardworking staff from across NLM who have taken advantage of these training opportunities to advance the goal of building a workforce for data driven research and health, both at NLM and throughout the biomedical and health sciences information world.
Share with us and others how you are helping your staff apply data science skills in your organization—do you have any lessons learned?
Guest post by Martha Meacham, MA, MLIS, NNLM Project Director
It’s a time of transformation and growth for the NLM’s Network of the National Library of Medicine (the Network or NNLM). Throughout its 61-year history, the Network has excelled at reaching people in communities throughout the United States. Today, the Network comprises more than 8,800 academic health science libraries, hospital, and public libraries and community organizations. NNLM has endured because of its ability to adapt and respond to changes in support of its mission to advance the progress of medicine and improve public health by serving librarians, researchers, clinicians, and the public. Today, approximately 90% of the U.S. population lives in a county with at least one NNLM member, and 93% of minority populations in the U.S. lives in a county with at least one NNLM member.
Leveraging the strength and expertise of its member organizations, NNLM offers funding for community-based projects that improve access to health information, increase engagement with research and data, expand professional knowledge, and support outreach that promotes awareness and use of NLM resources in local communities. Through the extraordinary work done by Network staff, NNLM has successfully developed and demonstrated effective engagement strategies in communities across the country. For example, the recently funded project “Surviving COVID-19, In A Virtual World” partnered with a local beauty salon to educate, train, and inform the community about COVID-19 and ways to prevent contracting and spreading the disease. The “Combatting COVID-19 Misinformation with Health Literacy Microcontent” project involved working with a community-based organization to provide easily accessible, culturally appropriate “microcontent,” short-form imagery and video content that can be consumed in 10-30 seconds or less, aimed at dispelling misinformation around COVID-19 and vaccines. And the “Informacion para tu Salud en tu Casa” project involved working with a local non-profit organization to improve the health and wellbeing of the Hispanic community by providing health information and resources, and by connecting people to health services through community health workers.
Strengthening the Network’s reach and impact requires continued evaluation and improvement. This includes reassessment of NNLM’s Regional Medical Libraries (RMLs). RMLs coordinate the operations of regional and national programs, as well as ensure a continuity of quality service for core programs of the NNLM.
Over the years, NNLM has evolved from a maximum of ten institutions serving as RMLs, to the long held eight RMLs, and now to a new configuration of seven RMLs.
This reorganization reconfigures regions and reduces disparities between regions in two ways:
Total population served
Number of member libraries and organizations supported
Balancing the regional areas of coverage and populations served allows for deeper connections and greater impact. Also, in addition to our traditional library partners, NNLM membership has expanded to include a wider variety of community-based organizations including faith-based organizations and K-12 schools, among others. As of April 2021, nearly 40% (3,482) of the total number of NNLM member organizations are not what we would have traditionally called libraries. This diversity supports NNLM’s new goal to “advance health equity through information,” with a focus on serving underrepresented populations.
RMLs will meaningfully engage with current and future audiences to increase information access, prioritizing underrepresented populations, including those experiencing health disparities by nature of race and ethnicity, biological sex, gender identity or expression, and sexual orientation, cognitive and physical abilities, religious background or identification, socioeconomic status (past and current), education level, health literacy, and linguistic needs, geographic location including underrepresented populations from medically underserved areas, and other factors that create unequal access to health care. The Network is positioned to address health inequities that contribute to health disparities.
NLM is committed to addressing the challenge of health disparities and seeks new ways to provide understandable and trusted health information resources in a variety of ways to support diverse and underserved populations.
A number of creative projects are underway to support NNLM’s new goal.
For more than 60 years, NNLM has provided a trusted local platform for community outreach and engagement to promote health. As we look to the future, new possibilities and an agile approach will maintain, build upon, and grow this successful and valued program.
Martha Meacham is the Project Director of NNLM. Martha is a passionate advocate for improving the health of all through access to and understanding of health information.
Guest post by Dianne Babski, Associate Director for Library Operations at NLM
How does an organization like NLM build and deploy 21st century products and services to support a global user audience? I’d like to give you a behind the scenes glimpse into NLM’s ever-evolving operations, and how we continue to develop the health information resources that you know and love, such as MEDLINE/PubMed, Medical Subject Headings (MeSH), and MedlinePlus.
Agile Product Development
NLM continues to move towards agile product development and digital unification. Where we used to release enhancements and features once or twice a year, we now develop incrementally and release product enhancements frequently. NLM supports innovation in our workforce by empowering product owners to make data-driven decisions through usability reviews and analytics of features, page views, and user requests to inform future actions.
We encourage staff to ask, “Are we meeting users’ needs—now and into the future?”
We have seen the success of this approach in the rollout of DOCLINE, our interlibrary loan request routing system, and the redesign of PubMed. We are in the planning phase of modernizing our flagship clinical trials registry and repository, ClinicalTrials.gov, to deliver an improved user experience on an updated platform to accommodate growth and enhance efficiency. We also embarked on the recommendations of several studies to increase the automation of MEDLINE Indexing. This involves incorporating machine learning and computational algorithms to apply MeSH terms to PubMed citations. As a result, the time for MEDLINE citations to be searched as indexed with MeSH in PubMed will be dramatically reduced, and, more importantly, will better leverage NLM staff expertise around chemical and gene names to enhance discoverability.
Data-Driven and Data-Informed
NLM uses data to balance our portfolio of products and offerings. I like to use the analogy of thinning garden beds to make room for healthier and stronger plants. We created evaluation measures to review our products and services, which allow us to make data-driven and data-informed decisions to streamline, simplify, and optimize NLM’s portfolio of offerings.
One key principle is to consolidate information into fewer platforms for improved user experience, discoverability, and efficiency. Pruning our garden allows us to focus on products that are unique, high-quality, and trusted resources. I think we can all agree that it’s more difficult to find what you need when information is scattered and disparate. This has informed the retirement of some products that are no longer sustainable or have a succession plan, or low or declining usage. And while a product may no longer exist as a stand-alone product, we have ensured that data and information from those products are integrated into others, made available for download, or both. For example, by integrating Genetics Home Reference and GeneEd data, we enhanced and made MedlinePlus more robust.
Other agencies or organizations sometimes have equally sufficient information and resources available that duplicate efforts. For example, this is true for the resources held in our Disaster Information Management Research Center (DIMRC), which we have begun retiring by limiting updates to select resources, such as Disaster Lit. This resource is currently only updated with COVID-19-related information as the product (or data) transitions ownership to other organizations. Meanwhile, much of the grey literature from Disaster Lit will remain available in the Digital Collections or the NLM Bookshelf.
To help users navigate NLM collections, we are upgrading our Integrated Library System infrastructure with a cloud-based library services platform. The new platform will allow for better systems integration, collaborative functionality, and community features to keep pace with the data demands of a digital ecosystem and enable better distribution to libraries worldwide. Stay tuned for a new and improved Catalog!
A Common Data Language
As a standards organization, NLM designs and integrates products to make information Findable, Accessible, Interoperable, and Reusable (FAIR). Following the FAIR data principles, an interconnected ecosystem of biomedical data, tools and software enables faster research conclusions and resulting publication(s).
NLM’s goal is to link different but related digital research objects, such as articles, data sets, visualization tools, and predictive models, to advance discovery within our vast collection and resources beyond NLM. For example, in response to the global COVID-19 pandemic, we quickly processed provisional out-of-cycle codes and terms from terminology sources in UMLS, RxNorm, SNOMED CT, and VSAC, added new MeSH and supplemental concept records, and new COVID-19-related Common Data Elements (CDEs) in the NIH CDE Repository. NLM also convened a trans-NIH team to identify NIH-endorsed data elements. We are extremely proud of the role we played in accelerating the interoperability and discoverability of critical COVID-19-related information to help solve a global health crisis.
Looking ahead to January 2023, NIH will adopt a new NIH Policy for Data Management and Sharing, requiring NIH-funded researchers to prospectively submit a plan outlining how scientific data from their research will be managed and shared. In response, NLM developed the Dataset Metadata Model (DATMM), designed to describe biomedical research datasets to drive discoverability and re-use of shared research data.
NLM connects globally to a large and diverse mix of stakeholders both in public and private sectors. Our products and services—no matter how agile, digital, or interconnected—would be nothing without our valued users.
We intentionally aggregate diverse data and analytical tools into our collections to advance research on factors such as biological, genomic, social, behavioral, and environmental impacts on health, and characteristics such as sex, gender, age, race and ethnicity. Working with other standards development groups, we are actively involved in efforts to represent sex, gender, race, and social determinants of health in their resources. We develop reliable health information in visual ways that are accessible to broad audiences, including users with low literacy. For example, MedlinePlus offers a series of brief videos (in English and Spanish) covering several popular health topics, and maintains a Health Information in Multiple Languages Collection featuring more than 60 languages to support the information needs of a global audience.
In its 2021-26 funding cycle, the NLM-supported Network of the National Library of Medicine has a new goal to “advance health equity through information”, and will focus on serving underrepresented populations. NLM remains committed to addressing the challenge of health disparities and seeks new ways to provide understandable and trusted health information resources in a variety of ways to support a broad spectrum of users.
I hope this peek inside of NLM gives you a sense of the ways that our dedicated staff are striving to meet the digital demands of the 21st century. Using our strategic plan as a roadmap, we continue to evaluate and develop products with our diverse user base in mind, and recognize that sometimes we need to rethink, rebuild, and reduce our presentation structures.
We’d love to hear how you are reimagining your services. Until next time, may your garden of health and knowledge blossom this spring!
Dianne Babski is responsible for the overall management of one of NLM’s largest divisions, Library Operations, with more than 450 staff providing health information services to a global audience of health care professionals, researchers, administrators, students, historians, patients, and the public. She oversees budget, facilities, administration, and operations, including of a national network of more than 8,000 academic health science libraries, hospital and public libraries, and community organizations to improve access to health information.
Guest post by Martha Meacham, MA, MLIS, NNLM Project Director
The NLM’s Network of the National Library of Medicine (NNLM) has a long, successful history of promoting access to and education about high quality health information, improving the health and health literacy of all. The COVID-19 pandemic may have changed how we approach our work, but our goals and successes have not changed. Adaptability, without sacrificing the quality and impact of our programs, is at our core. We’ve discovered new possibilities and engaged communities in new ways. These are just a few stories from across NNLM.
Even before the onset of the COVID-19 pandemic, NNLM had expertise conducting virtual programming due to the hundreds of events and classes offered each year. NNLM’s expertise was leveraged to expand our programming by transforming in-person events to virtual experiences, including virtual trainings, book clubs, and online symposiums addressing misinformation about COVID-19 — all with the goal of continuing to engage with communities.
Kiri Burcat, a Data and Evaluation Coordinator with the NNLM says, “NNLM was well-equipped for the COVID-19 work environment. With our regional/national collaborative model, we were used to video conferencing, long-distance collaboration, and online learning. While I hope to see my colleagues across the country in-person again soon, I also hope that this experience will lead us to experiment with different ways to keep online learning fresh and engaging.”
An attendee from one of NNLM’s virtual book club events shared how adapting to a virtual format provided different opportunities for the community, “I was at the Hood River Public Library and saw a book that caught my eye being given out at the COVID-safe space in the lobby. I learned about the upcoming author talk and the NNLM’s role in this effort. I was so impressed. I sent out a notice to my friends and encouraged them to join me in sharing the information on the book’s availability and the January 14th livestream event on Facebook and other social media to help reach more people. How inspirational! Thank you!”
Even with the restrictions of COVID-19, NNLM enabled its partners to continue their community outreach and engagement efforts. Cara Burton, System Director, Eastern Shore Public Library, Accomac, VA, highlighted, “The free print materials are very helpful in our outreach to poor, rural areas. For example, [NNLM-selected precision medicine] materials were distributed during COVID-19 in library packets at the public school free-meal pick-up sites. The NNLM staff did great outreach.”
NNLM coordinators across the country work closely with partners and member organizations to create and maintain high-quality work. “I have been incredibly inspired by the tenacity and innovation of our members who had to reinvent their organizations and services all at once to provide for their communities — and did so with excellence,” says Network Engagement Coordinator, Nancy Patterson.
The COVID-19 pandemic revealed opportunities for education on new topics and brought attention to needed skills. For example, a group within the NNLM organized a webinar series, Identifying and Combating Health Misinformation, featuring expert speakers who discussed various aspects of online health misinformation, how to identify it, and how to help curb its spread. After attending an event, one participant noted, “All of it was helpful. It will assist in better educating others on vaccines, the importance, and ways to know what is and what is not misleading information.” Another attendee noted that a benefit of the class was, “Learning about various ways misinformation can be spread. and learning about ways to stop the spread of misinformation.”
In another example of unique and timely programming, Liz Waltman, NNLM Outreach, Education, and Communications Coordinator notes, “The NNLM has had the opportunity to highlight the work our members are doing at this time. In particular, the webinar about evaluating information during COVID-19 was well attended and received.”
NNLM maintains its commitment to providing high quality educational and engagement opportunities for medical librarians and other professionals. Miso Lee, a data analyst with the University of Texas Medical Branch in Galveston, TX writes, “I am really grateful for [the] professional development award that allowed me to get the training I need. Second, I like informational webinars, particularly those related to COVID-19. I learned several creative ideas from other organizations.”
The resilience and adaptability of NNLM, founded on its unique expertise and experience, enabled this network of more than 8,000 academic health science libraries, hospital and public libraries, and community organizations to stretch, grow, and keep NLM relevant to communities, including medically underserved communities. Looking forward, NNLM will nurture the partnerships and approaches it has gained from this experience as it continues to expand and deepen NLM’s presence in communities across the country.
COVID-19 certainly has brought about changes and challenges, but through the great efforts of the NNLM staff and the wonderful work of its members and community partners, we remain strong and dedicated in these times.
Check out this upcoming series exploring the impact of COVID-19 and sign up here.
Martha Meacham is the Project Director of NNLM. Martha is a passionate advocate for improving the health of all through access to and understanding of health information.
Guest post by Rebecca Williams, PharmD, MPH, acting director of ClinicalTrials.gov at the National Library of Medicine, National Institutes of Health.
In 2019, NLM introduced a multi-year effort to modernize ClinicalTrials.gov, the world’s largest publicly accessible database of privately and publicly funded clinical trials. This effort was launched with a commitment to engage with and serve the millions of users who rely on this essential resource — with a focus on delivering an improved user experience on an updated platform that will accommodate growth and enhance efficiency.
In keeping with that promise, NLM has embarked on several stakeholder activities as part of the roadmap for modernization that we want to highlight in this post. We will also continue to share opportunities for involvement and invite you to join us for an upcoming webinar on February 18, 2021 at 3 pm ET to learn more about our modernization efforts.
Early in the process, our modernization team reached out to stakeholders through a request for information (RFI) to solicit input on topics around website functionality, information submission processes, and use of data standards. We received nearly 270 responses, which were summarized and discussed during a virtual interactive public meeting held in April 2020, and attended by nearly 400 participants. This robust feedback from the stakeholder community, along with input gathered from leaders and stakeholders across the NIH, provided the foundation to identify high-impact user-driven goals and set priorities. Currently, to advance the crucial next step of setting modernization goals and priorities, the team is working closely with NLM’s Board of Regents Public Service Working Group on ClinicalTrials.gov Modernization. Members in this group represent a range of stakeholder perspectives and provide input to ensure the continued integrity and utility of ClinicalTrials.gov.
Feedback received in response to last year’s RFI revealed several themes for modernizing ClinicalTrials.gov including ways to improve the management of search results, study records, and plain language information for the website, as well as enhancements to support structured and unstructured data, the quality control review process, and workflow management for information submission.
Three key external stakeholder groups were identified and consist of diverse individuals that share common interests and goals, but also have unique circumstances and needs. Feedback received from our engagement efforts, as well as the working group, reinforce the importance of being able to serve all stakeholders, including:
Data providers, who are responsible for submitting, updating, and managing clinical trial registration and results information through the ClinicalTrials.gov submission system, also known as the Protocol Registration and Results System (PRS). The clinical trial information provided by this stakeholder group is listed on ClinicalTrials.gov for use by others.
2.Patients and their advocates, who find and use information about clinical trials for themselves or others through ClinicalTrials.gov.
3. Data researchers, who use clinical trial information to study the clinical research enterprise, such as detecting trends in research and gaps in medical knowledge, identifying trials for use in systematic reviews and meta-analyses, and validating reported outcome measures and study design through ClinicalTrials.gov or its application programming interface, or API.
We aim to ensure that:
Clinical trial information is current, complete, and reliable.
All users can easily find and use information about clinical trials.
Clinical trial information, resources, and tools provide value to the research ecosystem.
NLM is employing a user-centered design approach to modernization. This includes user feedback loops and prototypes to elicit user input on new features in an iterative manner. A new version of the ClinicalTrials.gov website will be released first, in parallel to the current website, to obtain maximum user input. Work on the website began with extensive planning and research and has since shifted into implementation that includes building the technical foundation in a cloud-based platform; designing key elements for the homepage, search results, and study records; and testing approaches for the continuous improvement of search results. We expect to release the new website for broad user testing by Fall 2021. Work related to the PRS is in the planning and research stage, with prototype development underway.
As ClinicalTrials.gov and the PRS undergo “construction,” be assured that one of the core principles guiding NLM’s ClinicalTrials.gov modernization effort is to minimize disruption to our users while continuing to deliver improved services to maximize the value of this resource to the people who rely on it.
We are also developing continuity approaches for the PRS, management portal, and website to ensure a smooth transition to updated features. We will continue to provide stakeholders with updates and a timeline of activities as they become available.
Stay Informed and Involved
NLM is committed to keeping you informed throughout this modernization effort. We invite you to join us for an upcoming webinar onFebruary 18, 2021 at 3 pm ET where you will receive additional updates on our modernization efforts and learn about future opportunities for you and your communities to be involved. To register, please visit the ClinicalTrials.gov webinar registration page.
Rebecca Williams, PharmD, MPH, oversees the ClinicalTrials.gov program. Her research interests involve improving the quality of reporting of clinical research and evaluating the clinical research enterprise.