Progress Towards a Modernized ClinicalTrials.gov

Guest post by Rebecca Williams, PharmD, MPH, acting director of ClinicalTrials.gov at the National Library of Medicine, National Institutes of Health.

In 2019, NLM introduced a multi-year effort to modernize ClinicalTrials.gov, the world’s largest publicly accessible database of privately and publicly funded clinical trials. This effort was launched with a commitment to engage with and serve the millions of users who rely on this essential resource — with a focus on delivering an improved user experience on an updated platform that will accommodate growth and enhance efficiency.

In keeping with that promise, NLM has embarked on several stakeholder activities as part of the roadmap for modernization that we want to highlight in this post. We will also continue to share opportunities for involvement and invite you to join us for an upcoming webinar on February 18, 2021 at 3 pm ET to learn more about our modernization efforts.

Starting Out

Early in the process, our modernization team reached out to stakeholders through a request for information (RFI) to solicit input on topics around website functionality, information submission processes, and use of data standards. We received nearly 270 responses, which were summarized and discussed during a virtual interactive public meeting held in April 2020, and attended by nearly 400 participants. This robust feedback from the stakeholder community, along with input gathered from leaders and stakeholders across the NIH, provided the foundation to identify high-impact user-driven goals and set priorities. Currently, to advance the crucial next step of setting modernization goals and priorities, the team is working closely with NLM’s Board of Regents Public Service Working Group on ClinicalTrials.gov Modernization. Members in this group represent a range of stakeholder perspectives and provide input to ensure the continued integrity and utility of ClinicalTrials.gov.

Serving Stakeholders

Feedback received in response to last year’s RFI revealed several themes for modernizing ClinicalTrials.gov including ways to improve the management of search results, study records, and plain language information for the website, as well as enhancements to support structured and unstructured data, the quality control review process, and workflow management for information submission.

Three key external stakeholder groups were identified and consist of diverse individuals that share common interests and goals, but also have unique circumstances and needs. Feedback received from our engagement efforts, as well as the working group, reinforce the importance of being able to serve all stakeholders, including:

  1. Data providers, who are responsible for submitting, updating, and managing clinical trial registration and results information through the ClinicalTrials.gov submission system, also known as the Protocol Registration and Results System (PRS). The clinical trial information provided by this stakeholder group is listed on ClinicalTrials.gov for use by others.

2.Patients and their advocates, who find and use information about clinical trials for themselves or others through ClinicalTrials.gov. 

3. Data researchers, who use clinical trial information to study the clinical research enterprise, such as detecting trends in research and gaps in medical knowledge, identifying trials for use in systematic reviews and meta-analyses, and validating reported outcome measures and study design through ClinicalTrials.gov or its application programming interface, or API.

We aim to ensure that:

  1. Clinical trial information is current, complete, and reliable.
  2. All users can easily find and use information about clinical trials.
  3. Clinical trial information, resources, and tools provide value to the research ecosystem.

Looking Ahead

NLM is employing a user-centered design approach to modernization. This includes user feedback loops and prototypes to elicit user input on new features in an iterative manner. A new version of the ClinicalTrials.gov website will be released first, in parallel to the current website, to obtain maximum user input. Work on the website began with extensive planning and research and has since shifted into implementation that includes building the technical foundation in a cloud-based platform; designing key elements for the homepage, search results, and study records; and testing approaches for the continuous improvement of search results. We expect to release the new website for broad user testing by Fall 2021. Work related to the PRS is in the planning and research stage, with prototype development underway.

As ClinicalTrials.gov and the PRS undergo “construction,” be assured that one of the core principles guiding NLM’s ClinicalTrials.gov modernization effort is to minimize disruption to our users while continuing to deliver improved services to maximize the value of this resource to the people who rely on it.

We are also developing continuity approaches for the PRS, management portal, and website to ensure a smooth transition to updated features. We will continue to provide stakeholders with updates and a timeline of activities as they become available.

Stay Informed and Involved

NLM is committed to keeping you informed throughout this modernization effort. We invite you to join us for an upcoming webinar on February 18, 2021 at 3 pm ET where you will receive additional updates on our modernization efforts and learn about future opportunities for you and your communities to be involved. To register, please visit the ClinicalTrials.gov webinar registration page.

Rebecca Williams, PharmD, MPH, oversees the ClinicalTrials.gov program. Her research interests involve improving the quality of reporting of clinical research and evaluating the clinical research enterprise.

Celebrating 20 Years of ClinicalTrials.gov and Looking to the Future

Guest post by Rebecca Williams, PharmD, MPH, acting director of ClinicalTrials.gov at the National Library of Medicine, National Institutes of Health.

As ClinicalTrials.gov celebrates its 20th anniversary on February 29, 2020, we’re asking for your input on how it can best continue to serve your needs for many more years to come.

ClinicalTrials.gov is the world’s largest public clinical research registry and results database, giving patients, families, health care providers, researchers, and others easy access to information on clinical studies relating to a wide range of diseases and conditions. This online resource, which has more than 145,000 unique visitors every day, is operated by NLM and makes available information provided directly by the sponsors and investigators conducting the research.

NLM has launched an effort to modernize ClinicalTrials.gov to deliver an improved user experience on an updated platform that will accommodate growth and enhance efficiency. Creating a roadmap for modernization requires feedback from a wide array of stakeholders on how to continue serving, balancing, and prioritizing their varied information needs. These stakeholders include sponsors and investigators who submit clinical trial information to the site, academic institutions, nonprofit and advocacy organizations, government agencies, and the public, all of whom can access and use the information that ClinicalTrials.gov contains free of charge.

To obtain timely, detailed, and actionable input, we have issued a Request for Information (RFI) to solicit comments on the following topics: website functionality, information submission processes, and use of data standards.

Recognizing that ClinicalTrials.gov supports a network of stakeholders who contribute to, and rely on, clinical research, our aim is to understand how the system can better support this network and to identify opportunities for improving its compatibility with existing clinical trial management tools and processes. It is important to note that this RFI focuses on the functionality of ClinicalTrials.gov and is not intended to modify existing legal and policy requirements for clinical trial registration and results submission.

Over its 20-year history, ClinicalTrials.gov has helped shape the way in which clinical trial information is made transparent and discoverable to the public (see figure 1). In 2000, sponsors and investigators began submitting structured summaries of clinical trial protocols for the public to view. Over time, new policies and laws reinforced this practice, and ClinicalTrials.gov now contains over 320,000 study listings, with 56,000 studies currently seeking participants.

In 2008, ClinicalTrials.gov added its results database for sponsors and investigators to share summary results after trial completion. There are now over 40,000 results summaries posted on ClinicalTrials.gov, providing the public with timely access to information that may not be available in the peer-reviewed literature.

Figure 1: Total number of posted study records per year on ClinicalTrials.gov and timeline of major events from 1997 to 2019

Sharing information throughout the life cycle of a clinical trial (see figure 2) supports conduct of a landscape analysis prior to conducting new research and advances important public health goals, including supporting people who are looking to participate in clinical research, tracking the progress of clinical trials, allowing for the evaluation of the integrity of reported research, and providing more complete clinical trial information to help inform patient care. The modernization effort currently underway builds on the solid foundation established during the last 20 years.

Figure 2: Role of ClinicalTrials.gov in supporting use and sharing of information throughout the study life cycle

Share Your Feedback!

Responses to the RFI must be received by March 14, 2020. We expect a wide range of comments and are taking steps to manage and share the feedback. We will summarize the responses during a public meeting on April 30 on the main campus of the National Institutes of Health in Bethesda, Maryland, that will also be accessible by webcast. Details on the meeting will be available soon. In addition, we are engaging the NLM Board of Regents to provide input as we develop a roadmap for modernization, including establishing priorities and identifying the roles that various stakeholders might play in modernizing ClinicalTrials.gov.

Want to Learn More?

To learn more about the RFI and how to share your feedback, please join us for a webinar on January 22. We look forward to working with you to learn more about — and consider how to meet — your needs as we embark on this multiyear modernization effort.

Photo of Rebecca Williams, PharmD, MPH

Rebecca Williams, PharmD, MPH, oversees the ClinicalTrials.gov program. Her research interests involve improving the quality of reporting of clinical research and evaluating the clinical research enterprise.

Engaging Users to Support the Modernization of ClinicalTrials.gov

Guest post by Rebecca Williams, PharmD, MPH, acting director of ClinicalTrials.gov at the National Library of Medicine, National Institutes of Health.

ClinicalTrials.gov is the largest public clinical research registry and results database in the world – providing patients, health care providers, and researchers with information on more than 300,000 clinical studies of a wide range of diseases and conditions. More than 145,000 unique visitors use the public website daily to find and learn about clinical studies, resulting in an average of 215 million pageviews each month.

Recognizing the value of ClinicalTrials.gov to millions of users, the Board of Regents of the National Library of Medicine (NLM) described in the 2017-2027 strategic plan the importance of ensuring the long-term sustainability of this resource. NLM is committed to this goal and aims to modernize ClinicalTrials.gov to deliver a modern user experience on a flexible, extensible, scalable, and sustainable platform that will accommodate growth and enhance efficiency.

We are undertaking this effort to make ClinicalTrials.gov an even more valuable resource with a renewed commitment to engage with and serve the people who rely on it.

These users include the sponsors and investigators who submit clinical trial information for inclusion on the site through the submission portal. They also include patients, health care providers, and researchers who access listed information on ClinicalTrials.gov, whether directly or indirectly through other sites and services that use the ClinicalTrials.gov application programming interface.

Over the past several years, we have conducted testing with users and have already made some improvements in response to this feedback. With modernization, we will continue to support key functions identified by users of ClinicalTrials.gov while also seeking ways to make it an even more valuable resource.

To continue the modernization process, we are now seeking broader engagement with users to further help us determine how to evolve ClinicalTrials.gov. We are spending this summer looking inward by engaging our fellow National Institutes of Health Institutes and Centers to understand how ClinicalTrials.gov could better help in fulfilling NIH’s goals of clinical trial stewardship and transparency

This fall, we plan to expand our reach outward and are proposing to establish a working group of the NLM Board of Regents to focus on the modernization of ClinicalTrials.gov. This working group will provide a transparent forum for communicating and receiving input about efforts to enrich and modernize ClinicalTrials.gov. We want to ensure that we understand and consider changing needs while simultaneously maximizing the value of the growing amount of available information and preserving the integrity of ClinicalTrials.gov as a trusted resource.

We’ve already taken some steps to be more proactive in communicating with our users. We just launched “Hot Off the PRS!” (sign up to receive email announcements), a new informational bulletin for users of the ClinicalTrials.gov Protocol Registration and Results System (PRS). These updates provide timely announcements about new PRS features, relevant regulations (42 CFR Part 11) and policies, and information about other offerings such as the PRS Guided Tutorials (BETA), a new training resource with step-by-step instructions for submitting results information.

We’re excited about how greater user engagement will enrich and modernize ClinicalTrials.gov, improving its value for everyone throughout the clinical research lifecycle.

Please let us know what else we can do to make ClinicalTrials.gov the best it can be.

Photo of Rebecca Williams, PharmD, MPH

Rebecca Williams, PharmD, MPH, oversees the technical, scientific, policy, regulatory and outreach activities related to the operation of ClinicalTrials.gov. Her research interests relate to improving the quality of reporting of clinical research and evaluating the clinical research enterprise.

Can “Nudging” Help?: Improving Clinical Trial Access Using Artificial Intelligence for Standardization

Guest post by Presidential Innovation Fellows Justin Koufopoulos and Gil Alterovitz, PhD.

Getting into a clinical trial is challenging for patients. Researchers estimate that only 5% of patients eligible to participate in a cancer clinical trial actually take part in one.  Many factors impact this statistic, including how findable and accessible is information about the clinical trials.

Patients often learn about clinical trials from their doctors or through patient advocacy groups like the American Cancer Society. They then typically search for trials on the internet, often ending up on websites like the NIH-run ClinicalTrials.gov or trials.cancer.gov.

Once on these websites, patients still face challenges to access. Prime among them: what search terms to use to find relevant trials.

The terms a patient or doctor uses may not match how researchers running a trial describe the focus of their study, for example “breast cancer” vs. “ductal carcinoma.” While the NIH clinical trials databases track synonyms and work to make the proper matches, users cannot escape this recurring mismatch in language that challenges access.

This challenge becomes even more pronounced with clinical trial eligibility criteria. These criteria describe who can and cannot participate in a study. For example, an eligibility criterion might be “age 18 years or older” or “confirmed breast lesions that can proceed to removal without chemotherapy.” While a computer can easily match a patient to the first criterion, the second involves many more concepts that are harder to separate, understand, and match.

Artificial intelligence can be part of the solution, particularly “machine learning,” which leverages data to teach a program how to make predictions on chosen topics.

Various technology companies have already used machine learning to address language translation problems. As a result, computers can now translate English to Japanese with few errors, and speech-to-text applications can translate human speech to computer inputs and can even reply.

We adopted a similar, albeit scaled back, approach to translate diverse clinical trials eligibility criteria into standardized and structured language. We also drew inspiration from writing tools that help writers improve their text’s readability and grammar.

Instead of highlighting repeated words or sentences in the passive voice, our prototype nudges researchers toward writing eligibility criteria in a way more easily translated by machine. It offers feedback and suggestions, almost like an English language tutor, and proposes alternative ways to write the criteria that would make them more standard and eventually, more translatable.

Sample Word text with track changes

screen shot from within the eligibility criteria normalizer showing alternate phrasings for a sample criterio
We drew inspiration from revision tracking and grammar-type tools to design our standardization tool for researchers.

This shift toward more standardized language can make it easier to match content across databases, such as matching a list of patients with a set of conditions and prior treatments.

The prototype also helps researchers understand the consequences of their word choices. It looks at previous studies with similar eligibility criteria and notes how many participants they recruited. Additionally, input from consensus-based standards may also be presented.  While not a perfect metric for inclusiveness, this feedback shows someone running a study how their word choices compare to others and the potential impact of those choices on their study’s overall success.

Research by academic psychologists has shown that nudging works in a wide variety of settings. To the best of our knowledge, this is the first time a nudge has been used to coach researchers, but these nudges are not requirements. Researchers can still write their eligibility criteria in the way they think makes the most sense. However, by moving researchers toward standardized phrasings, our prototype can help computers match patient characteristics with eligibility criteria and potentially get more eligible patients into clinical trials.

More work is needed before we can fully implement our tool and test at scale, but we are making progress. We recently completed a pilot study with non-federal groups to determine whether the structured data (so, not the nudging agent but the data our tool learns from) could be used to create tools to help with clinical trials access. Our findings were positive, confirming that private industry and academia need more data like ours for building artificial intelligence tools. The work was featured by the White House on AI.gov as an example for “Industries of the Future.”

The Health Sprint piloting effort included physicians and patient advocates as well as data stewards and experts in the relevant domain areas from within government. For example, Rick Bangs, MBA, PMP, a patient advocate, has worked with various organizations including the National Cancer Institute and the ClinicalTrials.gov development team. Regarding clinical trial matching, Bangs noted, “The solution here will require vision, and that vision will cross capabilities that no one supplier will individually have.”

Next up, we need to evaluate whether this tool helps researchers write eligibility criteria in the “real world,” where all innovations must live.

headshot of Justin KoufopoulosJustin Koufopoulos is a Presidential Innovation Fellow and product manager working to making clinical research more patient-centered. He has worked with the White House, CIO Council, National Library of Medicine, General Services Administration, Department of Commerce, and Veterans Administration on issues ranging from internet access to artificial intelligence.

headshot of Gil AlterovitzGil Alterovitz, PhD, FACMI, is a Presidential Innovation Fellow who has worked on bridging data ecosystems and artificial intelligence at the interface of several federal organizations, including the White House, National Cancer Institute, General Services Administration, CIO Council, and Veterans Administration.


The Presidential Innovation Fellowship brings together top innovators and their insights from outside of government, including the private sector, non-profits, and academia. Their insights are brought to bear on some of the most challenging problems within government and its agencies. The goal is to challenge existing paradigms by rethinking problems and leveraging novel, agile approaches. PIF was congressionally mandated under HR 39, the Tested Ability to Leverage Exceptional National Talent (TALENT) Act. The program is administered as a partnership between the White House Office of Science and Technology Policy, the White House Office of Management and Budget, and the General Services Administration.

ClinicalTrials.gov Moves Toward Increased Transparency

Guest post by Kevin M. Fain, JD, MPH, DrPH, Senior Advisor for Policy and Research, ClinicalTrials.gov.

ClinicalTrials.gov is the largest public clinical research registry and results database in the world—and  the most heavily used. As of today, it contains registration information for more than 260,000 studies in 202 different countries and results information on more than 29,000 of those studies. Each week, the content grows by approximately 560 new registrations and 110 new results submissions. The system averages more than 162 million page views per month and 93,000 unique visitors daily.

ClinicalTrials.gov enables users to: (1) search for clinical trials of drugs, biologics, devices, and other interventions; (2) obtain summary information about these studies (e.g., purpose, design, and facility locations); (3) track the progress of a study from initiation to completion; and (4) obtain summary results, often before they are published elsewhere.

In addition, the unique ClinicalTrials.gov identifier assigned to each registered trial (commonly referred to as the “NCT Number”) has become the de facto standard for referencing trials and is widely and routinely used in medical journal articles, MEDLINE citations, and the mass media.

Federal law underlies the ClinicalTrials.gov database requirements and content. NIH launched the database in 2000 after the Food and Drug Administration Modernization Act of 1997.  The FDA Amendments Act of 2007 then expanded the database’s scope and purpose by requiring registration and results reporting for certain clinical trials of FDA-regulated drugs, biological products, and medical devices. Importantly, the 2007 law included legal consequences for noncompliance, including civil monetary penalties.

More recently, in an effort to make information about clinical trials more widely available to the public, the US Department of Health and Human Services issued a final rule in September 2016 that specifies requirements for registering certain clinical trials and submitting summary results information to ClinicalTrials.gov. The rule’s final form was shaped by over 900 public comments.

The new rule, which became effective one year ago (January 18, 2017), clarifies and expands the reporting requirements for clinical trials, including trial results for drug, biologic, and device products not approved by FDA. At the same time, NIH issued a policy establishing the expectation that all investigators conducting clinical trials funded in whole or in part by NIH will ensure these trials are registered at ClinicalTrials.gov and that results information for these trials is submitted to ClinicalTrials.gov.

The expanded reporting requirements are expected to yield important scientific, medical, and public health benefits—from improving the clinical research enterprise itself to maintaining the public’s trust in clinical research. Having access to complete study results, including negative or inconclusive data, can help counteract publication bias, reduce duplication in research, improve the focus and design of future studies, and protect patients from undue risk or ineffective interventions. That additional information, in the context of other research, can also help inform health care providers and patients regarding medical decisions.

As a repository for study results, ClinicalTrials.gov helps deliver those benefits.

Recent research indicates that the results of many clinical trials—including those funded by NIH—are never published. And even when results are published, they can be limited, focusing on findings of most interest rather than all outcomes. In contrast, studies have found results reported in ClinicalTrials.gov are more complete than in the published literature. The new reporting requirements are expected to strengthen that characteristic and enhance the benefits ClinicalTrials.gov brings.

It is important to understand that listing a study on ClinicalTrials.gov does not mean it has been evaluated by the US Federal Government. The ClinicalTrials.gov website emphasizes this point for the public through prominent disclaimer statements, including one on the importance of discussing any clinical trial with a health care provider before participating.

ClinicalTrials.gov allows for the registration of any human biomedical study that conforms with prevailing laws and regulations, including an indication that recruiting studies were approved by an ethics review committee. As a result, the database is more comprehensive, which can better serve the public in critical ways. For example, potential participants can see the full range of studies being conducted, not just those funded or sponsored by NIH. Ethics committees, funders, and others can also view the wider scope of studies, which can help them more effectively oversee new research.

Aside from legislative and policy changes, ClinicalTrials.gov has also focused on enhancing the site’s usability, addressing design and layout issues and improving the ability to search, display, and review information about the studies registered on the site. The latest set of updates, released last month, included new search options (such as by recruitment status and distance from a geographic location), refinements to the display of search results, and additional information regarding study results and key record dates.  These changes, plus those brought about by the final rule, will help maximize the value of clinical trials, and by extension, advance knowledge and improve health.

From finding trials actively recruiting participants to identifying new experimental drugs or device interventions to analyzing study design and results, ClinicalTrials.gov delivers key benefits to patients, clinicians, and researchers and puts into action NIH’s core mission: turning discovery into health. It also reflects one more way NLM makes medical and health information available for public use and patient health.

headshot of Kevin FainKevin Fain, JD, MPH, DrPH, has served as senior advisor for policy and research at ClinicalTrials.gov since 2015. He was an attorney with the FDA from 1995-2010, specializing in clinical trial and drug regulatory matters. He earned his doctorate in epidemiology from Johns Hopkins University in 2015.