Artificial Intelligence, Imaging, and the Promising Future of Image-Based Medicine

In mid-October I gave the NLM Research in Trustable, Transparent AI for Decision Support keynote speech to the 50th Institute of Electrical and Electronics Engineers (IEEE) Applied Imagery Pattern Recognition conference in Washington, D.C. (virtually, for me). The IEEE continues to advance new topics in applied image and visual understanding, and the focus this year was to explore artificial intelligence (AI) in medicine, health care, and neuroscience.

To prepare for my talk, I reviewed our extramural research portfolio so I could highlight current research on these topics. NLM’s brilliant investigators are using a range of machine learning and AI strategies to analyze diverse image types. Some of the work fosters biomedical discovery; other work is focused on creating novel decision support or quality improvement strategies for clinical care. As I did with the audience at IEEE, I’d like to introduce you to a few of these investigators and their projects.

Hagit Shatkay and her colleagues from the University of Delaware direct a project titled Incorporating Image-based Features into Biomedical Document Classification. This research aims to support and accelerate the search for biomedical literature by leveraging images within articles that are rich and essential indicators for relevance-based searches. This project will build robust tools for harvesting images from PDF articles and segment compound figures into individual image panels, identify and investigate features for biomedical image-representation and categorization of biomedical images, and create an effective representation of documents using text and images grounded in the integration of text-based and image-based classifiers.

Hailing from the University of Michigan, Jenna Wiens leads a project called Leveraging Clinical Time Series to Learn Optimal Treatment of Acute Dyspnea. Managing patients with acute dyspnea is a challenge, sometimes requiring minute-to-minute changes in care approaches. This team will develop a novel clinician-in-the-loop reinforcement learning (RL) framework that analyzes electronic health record (EHR) clinical time-series data to support physician decision-making. RL differs from the more traditional classification-based supervised learning approach to prediction; RL “learns” from evaluating multiple pathways to many different solution states. Wiens’ team will create a shareable, de-identified EHR time-series dataset of 35,000 patients with acute dyspnea and develop techniques for exploiting invariances (different approaches to the same outcome) in tasks involving clinical time-series data. Finally, the team will develop and evaluate an RL-based framework for learning optimal treatment policies and validating the learned treatment model prospectively.

Quynh Nguyen from the University of Maryland leads a project called Neighborhood Looking Glass: 360 Degree Automated Characterization of the Built Environment for Neighborhood Effects Research. Using geographic information systems and images to assemble a national collection of all road intersections and street segments in the United States, this team is developing informatics algorithms to capture neighborhood characteristics to assess the potential impact on health.

Corey Lester from the University of Michigan leads a multidisciplinary team using machine intelligence in a project titled Preventing Medication Dispensing Errors in Pharmacy Practice with Interpretable Machine Intelligence. Machine intelligence is a branch of AI distinguished by its reliance on deductive logic, and the ability to make continuous modifications based in part on its ability to detect patterns and trends in data. The team is designing interpretable machine intelligence to double-check dispensed medication images in real-time, evaluate changes in pharmacy staff trust, and determine the effect of interpretable machine intelligence on long-term pharmacy staff performance. More than 50,000 images are captured and put through an automated check process to predict the shape, color, and National Drug Code of the medication product. This use of interpretable machine intelligence methods in the context of medication dispensing is designed to provide pharmacists with confirmatory information about prescription accuracy in a way that reduces cognitive demand while promoting patient safety.

Alan McMillan from the University of Wisconsin-Madison and his team are examining how image interpretation can improve noisy data in a project called Can Machines be Trusted? Robustification of Deep Learning for Medical Imaging. Noisy data is information that cannot be understood and interpreted correctly by machines (such as unstructured text). While deep learning approaches (methods that automatically extract high-level features from input data to discern relationships) to image interpretation is gaining acceptance, these algorithms can fail when the images themselves include small errors arising from problems with the image capture or slight movements (e.g., chest excursion in the breathing of the patient). The project team will probe the limits of deep learning when presented with noisy data with the ultimate goal of making the deep learning algorithms more robust for clinical use.

In the work of Joshua Campbell’s team at Boston University, the images emerge at the end of the process to allow for visualization of large-scale datasets of single-cell data. The project, titled Integrative Clustering of Cells and Samples Using Multi-Modal Single-Cell Datauses a Bayesian hierarchical model developed by the team to perform bi-clustering of genes into modules and cells into subpopulations. The team is developing innovative models that cluster cells into subpopulations using multiple data types and cluster patients into subgroups using both single-cell data and patient-level characteristics. This approach offers improvements over discrete Bayesian hierarchical models for classification in that it will support multi-modal and multilevel clustering of data.

Several things struck me as I reviewed these research projects. The first was a sense of excitement over the engagement of so many smart young people at the intersection of analytics, biomedicine, and technology. The second was the variety of types of images considered within each project. While one study explores radiological images, another study examines how image data types vary from figures in journal papers to pictures of the built environment and images of workflows in a pharmacy. Two of these studies use AI techniques to analyze the impact of the physical environment to better understand its influence on patient health and safety, and one study uses images as a visualization tool to better support inference of large-scale biomedical research projects. Images appear at all points of the research process, and their effective use heralds an era of image-based medicine. Let’s see what lies ahead!

Request for Public Comment: Seeking Input on Nationwide AI Research Resource Implementation Plan

Guest post by Lynne E. Parker, PhD, Director of the White House National Artificial Intelligence Initiative Office, and Erwin Gianchandani, PhD, National Science Foundation Senior Advisor for Translation, Innovation, and Partnerships

The White House Office of Science and Technology Policy and National Science Foundation are looking for your input to shape the work of the National Artificial Intelligence Research Resource (NAIRR) Task Force. This Task Force is taking on a critically important initiative – building an implementation plan for a national infrastructure that would democratize access to artificial intelligence (AI) research and development (R&D).  

As directed by Congress in the National AI Initiative Act of 2020, the Task Force is serving as a Federal advisory committee to help create a blueprint for the NAIRR, which is envisioned as a shared computing and data infrastructure that would provide AI researchers and students across all scientific disciplines with access to computational resources, high-quality data, educational tools, and user support. This capability would help make AI R&D accessible to all Americans by lowering the barriers to entry for traditionally underserved communities, institutions, and regions. It would also fuel innovation by making it easier than ever before for Americans to pursue bold, visionary applications for AI.

The Task Force will provide recommendations for establishing and sustaining the NAIRR, including technical capabilities, governance, administration, and assessment, as well as requirements for security, privacy, civil rights, and civil liberties. The Task Force will submit two reports to Congress presenting a comprehensive strategy and implementation plan: an interim report in May 2022 and final report in November 2022.

To get this right, we want to tap into the deep technical expertise of the community and bring in a range of perspectives. We invite you to submit a response to our Request for Information before the comment period closes on October 1, and ask that you spread the word. This effort could set us on the path to transform our nation’s ability to harness AI across fields of science and engineering and economic sectors, and your insights could help shape our approach.

We appreciate your contributions and look forward to receiving your input.

Dr. Parker is the Founding Director of the National Artificial Intelligence (AI) Initiative Office and Assistant Director of AI in the White House Office of Science and Technology Policy (OSTP).  In these roles, she leads national AI policy efforts and coordinates AI activities across the Federal agencies in support of the National AI Initiative.  Dr. Parker is a professor of computer science at the University of Tennessee, on assignment to OSTP. She received her PhD from the Massachusetts Institute of Technology and is a Fellow of the American Association for the Advancement of Science and Institute of Electrical and Electronics Engineers.

Dr. Gianchandani is the National Science Foundation (NSF) Senior Advisor for Translation, Innovation, and Partnerships. For six years, he was the NSF Deputy Assistant Director for Computer and Information Science and Engineering. In this role, he contributed to the leadership and management of NSF’s CISE directorate, including formulation and implementation of the directorate’s $1 billion annual budget, strategic and human capital planning, and oversight of day-to-day operations. He received a BS in computer science and MS and PhD degrees in biomedical engineering from the University of Virginia.

NIH Strategically, and Ethically, Building a Bridge to AI (Bridge2AI)

This piece was authored by staff across NIH that serve on the working group for the NIH Common Fund’s Bridge2AI program—a new trans-NIH effort to harness the power of AI to propel biomedical and behavioral research forward.

The evolving field of Artificial Intelligence (AI) has the potential to revolutionize scientific discovery from bench to bedside. The understanding of human health and disease has vastly expanded as a result of research supported by the National Institutes of Health (NIH) and others. Every discovery and advance in contemporary medicine comes with a deluge of data. These large quantities of data, however, still result in restricted, incomplete views into the natural processes underlying human health and disease. These complex processes occur across the “health-disease” spectrum over temporal scales – sub-seconds to years – and biological scales – atomic, molecular, cellular, organ systems, individual to population. AI provides the computational and analytical tools that have the potential to connect the dots across these scales to drive discovery and clinical utility from all of the available evidence.

A new NIH Common Fund program, Bridge to Artificial Intelligence (Bridge2AI), will tap into the power of AI to lead the way toward insights that can ultimately inform clinical decisions and individualize care. AI, which encompasses many methods, including modern machine learning (ML), offers potential solutions to many challenges in biomedical and behavioral research.

AI emerged in the 1960s and has evolved substantially in the past two decades in terms of its utility for biomedical research. The impact of AI for biomedical and behavioral research and clinical care derives from its ability to use computer algorithms to quickly find connections from within large data sets and predict future outcomes. AI is already used to improve diagnostic accuracy, increase efficiency in workflow and clinical operations, and facilitate disease and therapeutic monitoring, to name a few applications. To date, the FDA has approved more than 100 AI-based medical products.

AI-assisted learning and discovery is only as good as the data used to train it. 

The use of AI/ML modeling in biomedical and behavioral research is limited by the availability of well-defined data to “train” AI algorithms to learn how to recognize patterns within the data. Existing biomedical and behavioral data sets rarely include all necessary information as they are collected on relatively small samples and lack the diversity of the U.S. population. Data from a variety of sources are necessary to characterize human health, such as those from -omics, imaging, behavior, and clinical indicators, electronic health records, wearable sensors, and population health summaries. The data generation process itself involves human assumptions, inferences, and biases that must be considered in developing ethical principles surrounding data collection and use. Standardizing collection processes is challenging and requires new approaches and methods. Comprehensive, systematically generated and carefully collected data is critical to build AI models that provide actionable information and predictive power. Data generation remains among the greatest challenges that must be resolved for AI to have a real-world impact on medicine.

Bridge2AI is a bold new initiative at the National Institutes of Health designed to propel research forward by accelerating AI/ML solutions to complex biomedical and behavioral health challenges whose resolution lies far beyond human intuition. Bridge2AI will support the generation of new biomedically relevant data sets amenable to AI/ML analysis at scale; development of standards across multiple data sources and types; production of tools to accelerate the creation of FAIR (Findable, Accessible, Interoperable, Reusable) AI/ML-ready data; design of skills and workforce development materials and activities; and promotion of a culture of diversity and ethical inquiry throughout the data generation process.

Bridge2AI plans to support several Data Generation Projects and an Integration, Dissemination and Evaluation (BRIDGE) Center to develop best practices for the use of AI/ML in biomedical and behavioral research. For additional information, see NOT-OD-21-021 and NOT-OD-21-022. Keep up with the latest news by visiting the Bridge2AI website regularly and subscribing to the Bridge2AI listserv.

Top Row (left to right):
Patricia Flatley Brennan, RN, PhD, Director, National Library of Medicine
Michael F. Chiang, MD, Director, National Eye Institute
Eric Green, MD, PhD, Director, National Human Genome Research Institute
Bottom Row (left to right):
Helene Langevin, MD, Director, National Center for Complementary and Integrative Health
Bruce J. Tromberg, PhD, Director, National Institute of Biomedical Imaging and Bioengineering

AI is coming. Are the data ready?

The artificial intelligence (AI) revolution is upon us. You can barely read the paper, watch TV, or see a movie without encountering AI and how it promises to change society. In fact, last month, the President signed an executive order directing the US government to prioritize artificial intelligence in its research and development spending to help drive economic growth and benefit the American people.

Artificial intelligence refers to a suite of computer analysis methods—including machine learning, neural networks, deep learning models, and natural language processing—that can enable machines to function as if possessing human reasoning. With AI, computer systems ingest and analyze vast amounts of data and then “learn” through high-volume repetition how to do the task better and better, “reasoning” or “self-modifying” to improve the analytics that shape the outcome.

That learning process results in some pretty amazing stuff. In the health care field alone, AI can determine the presence or absence of abnormalities in clinical images, predict which patients are at risk for rare disorders, and detect irregular heartbeats.

To make all that happen requires data, massive amounts of data.

But like the computer-era quip, “garbage in, garbage out,” the data need to be good to yield valid analyses. What does “good” mean? Two things:

  • The data are accurate, truly representing the underlying phenomena.
  • The data are unbiased, i.e., the observations reflect the complete experience and no inherent errors were introduced anywhere along the chain from data capture to coding to processing.

As much as we’d like to think otherwise, we already know data are biased. Human genetic sequences drawn from studies of white males of Northern European descent do not adequately represent the genetic diversity within women or people from other parts of the globe. Image data generated by different X-ray machines might show slight variations depending upon how the machines were calibrated. Electrical pathways collected from neurological studies conducted as recently as a decade ago do not reflect the level of resolution possible today.

So, what can we do?

It doesn’t make sense to throw out existing data and start anew, but it can be misleading to apply AI to data known to be biased. And it can be risky. Bias in underlying data can result in algorithms that propagate the same bias, leading to inaccurate findings.

That’s why NLM is working to develop computational approaches to account for bias in existing data sets and why we’re investing in this line of research. In fact, we’re actively encouraging grant applications focused on reducing or mitigating gaps and errors in health data research sets.

I have confidence that researchers will crack the puzzle, but until then, let’s look at how the business intelligence community is approaching the issue.

Concerned with reducing the effect of biases in management decision-making, business intelligence specialists have identified strategies to help uncover patterns and probabilities in data sets. They pair these patterns with AI algorithms to create calibration tools informed by human judgment while taking advantage of the algorithms’ power. That same approach might work with biomedical data.

In addition, our colleagues in business now approach data analysis in ways that help detect bias and limit its impact. They:

  • invest more human resources in interpreting the results of AI analytics, not relying exclusively on the algorithms;
  • challenge decision makers to consider plausible alternative explanations for the generated results; and
  • train decision makers to be skeptical and to anticipate aberrant findings.

There’s no reason we can’t adopt that approach in biomedical research.

So, as you read and think more about the potential of artificial intelligence, remember that AI applications are only as good as the data upon which they are trained and built. Remember, too, that the results of an AI-powered analysis should only factor in to the final decision; they should not be the final arbiter of that decision. After all, the findings may sound good, but they may not be real, just an artifact of biased, imperfect data.

Models: The Third Leg in Data-Driven Discovery

Considering a library of models

George Box, a famous statistician, once remarked, “All models are wrong, and some are useful.”

As representations or approximations of real-world phenomena, models, when done well, can be very useful.  In fact, they serve as the third leg to the stool that is data-driven discovery, joining the published literature and its underlying data to give investigators the materials necessary to explore important dynamics in health and biomedicine.

By isolating and replicating key aspects within complex phenomena, models help us better understand what’s going on and how the pieces or processes fit together.

Because of the complexity within biomedicine, health care research must employ different kinds of models, depending on what’s being looked at.

Regardless of the type used, however, models take time to build, because the model builder must first understand the elements of the phenomena that must be represented. Only then can she select the appropriate modeling tools and build the model.

Tracking and storing models can help with that.

Not only would tracking models enable re-use—saving valuable time and money—but doing so would enhance the rigor and reproducibility of the research itself by giving scientists the ability to see and test the methodology behind the data.

Enter libraries.

As we’ve done for the literature, libraries can help document and preserve models and make them discoverable.

The first step in that is identifying and collecting useful models.

Second, we’d have to apply metadata to describe the models. Among the essential elements to include in such descriptions might be model type, purpose, key underlying assumptions, referent scale, and indicators of how and when the model was used.

screencapture with the DOI and RRIDs highlighted
The DOI and RRIDs in a current PubMed record.
(Click to enlarge.)

We’d then need to apply one or more unique identifiers to help with curation. Currently, two different schema provide principled ways to identify models: the Digital Object Identifier (DOI) and the Research Resource Identifier (RRID). The former provides a persistent, unique code to track an item or entity at an overarching level (e.g., an article or book).  The latter documents the main resources used to produce the scientific findings in that article or book (e.g., antibodies, model organisms, computational models).

Just as clicking on an author’s name in PubMed can bring up all the articles he or she has written, these interoperable identifiers, once assigned to research models, make it possible to connect the studies employing those models.  Effectively, these identifiers can tie together the three components that underpin data-driven discovery—the literature, the supporting data, and the analytical tools—thus enhancing discoverability and streamlining scientific communication.

NLM’s long-standing role in collecting, organizing, and making available the biomedical literature positions us well to take on the task of tracking research models, but is that something we should do?

If so, what might that library of models look like? What else should it include? And how useful would this library of models be to you?

Photo credit (stool, top): Doug Belshaw [Flickr (CC BY 2.0) | erased text from original]
%d bloggers like this: