Guest post by Kimberly Thomas, MPH, Strategic Evaluation Officer, National Library of Medicine (NLM) Office of Strategic Initiatives, and Alla Keselman, PhD, MA, Acting Chief, NLM Office of Engagement and Training (OET).
However beautiful the strategy, you should occasionally look at the results.
From scientific decision making to how infrastructure and operations support improved efficiencies, the context in which NLM works has dramatically shifted from “What?” to “So what?” and even “Now what?” That is, beyond the vast portfolio of products and services that NLM provides (‘What’), how do we seek to identify impact and benefit to the American people (‘So what’)? And in the face of observed outcomes and impacts (the results and effects of our products, services, and research), what should we do next? How do we learn from the data and information about our efforts to advance excellence and innovation at NLM (‘Now what’)?
Organizational Learning at NLM
Prioritizing and answering questions about the outcomes of our products, services, and research contributes to the evidence-base, which advances data-driven decision making across NLM. These decisions support our customers and demonstrate the value of responsible stewardship of the federal funds entrusted to us. Grounded in our core values, innovation and accountability have long been a cornerstone of NLM. In 2017, an Evaluation Task Force developed a framework to advance NLM in the application of web metrics and digital analytics tools and approaches and recommended NLM convene a committee to meet regularly about evidence-based activities. In 2021, the NLM Evaluation Coordinating Committee (NECC) was established to advance trans-NLM communication, collaboration, and coordination of activities related to the Evidence Act. Through the NECC, NLM now has an initial inventory of evidence bases that support initiatives, products, processes, and various programmatic activities such as training, outreach, and engagement.
We’ve amplified our approach to using data and information by engaging in thoughtful discussions with NLM Senior Leadership. How do they use data to inform their knowledge and decision-making priorities? In what areas do we need to increase evidence-based decision making across NLM? We’ve also initiated important discussions with our stakeholders such as the Board of Regents about how NLM might prioritize evidence-informed work, including the questions we ask about key priority efforts and progress toward the three goals of the NLM Strategic Plan 2017–2027. These discussions will be a critical and ongoing part of our strategy for organizational learning. In this way, NLM not only demonstrates systematic progress in priority areas, but we do so in a way that allows for continued strategic innovation and growth.
NLM Division of Library Operations: Evidence in Action
As one of many NLM examples, the Division of Library Operations (LO) employs a comprehensive approach to evidence-informed decision making by using data in multiple ways, from product design (e.g., web analytics and usability data) and indexing techniques (e.g., recall and precision data) to strategic planning.
One of LO’s core functions is to connect global audiences to biomedical data and resources through the Network of the National Library of Medicine (NNLM). NNLM is comprised of more than 8,000 members, which include public libraries, community colleges, community-based organizations, clinics, and more. The “what” of NNLM programming is free training opportunities and information resources as well as funding for members’ community outreach and capacity-building programs. The “so what” is the question of the impact of NNLM programming on members and their audiences. Measures of impact important to NLM include awareness of NLM and its offerings, trust in NLM resources and mission, the public’s health information literacy, and competencies and career trajectories of information professionals. To assess these measures, the NNLM National Evaluation Center (NEC), working jointly with LO Office of Engagement and Training, develops a strategic approach to evidence-building and creates instruments used across regions and programs that help standardize the NNLM evidence base. The “now what” is using the data obtained by the NEC to inform the process of continuous improvement such as enhancing training courses.
Looking Ahead
Organizational learning is about using evidence to improve outcomes, demonstrate impact, and inform decision making. It’s also about preserving and curating NLM’s institutional memory to ensure we maintain and build upon valuable knowledge, leveraging past experiences, successes, and lessons learned. In shifting our focus from “What?” to “So what?,” NLM continues to strengthen use of data for informed decision making while demonstrating our commitment to results-based accountability. What kinds of data and information do you use to measure outcomes, drive change, and foster innovation?
Kimberly Thomas, MPH
Strategic Evaluation Officer, NLM Office of Strategic InitiativesMs. Thomas is responsible for formulating and executing a framework for identifying, prioritizing, conducting, and coordinating programmatic evaluations to better understand alignment of NLM activities with its strategic goals and objectives. Prior to joining NLM, Kimberly spent most of her career at the Centers for Disease Control and Prevention using program evaluation and surveillance data to improve programmatic decision making and to promote data-driven outcomes. She holds a BS in Health Science from Georgia Southern University and an MPH in Social and Behavioral Sciences from the Morehouse School of Medicine.
Alla Keselman, PhD, MA
Acting Chief, NLM Office of Engagement and TrainingDr. Keselman is responsible for connecting users to NLM information and engaging medically underserved populations and communities. Prior to her current role, Dr. Keselman coordinated NLM efforts in evaluating the impact of its health information outreach and community engagement programs. She also conducted research into the public’s understanding of complex health concepts, scientific literacy, and health information provision outside clinical settings. Dr. Keselman holds a PhD in Human Cognition and Learning and an MA in Biomedical Informatics from Columbia University.