Optimizing Real-World Evidence Use at FDA, Workshop Insights
December 18, 2024
Introduction
Improved access to digital healthcare data of all types and the ability to rapidly analyze data has led to interest in the unique insights this data might yield to help us all live healthier lives. Real-world data (RWD) and the real-world evidence (RWE) derived from it have the potential to transform drug development, promising to bring down the cost and increase the diversity of study populations. As the volume and variability of RWD continues to grow, important issues regarding data quality, determining fit-for-use, and appropriate methods to analyze non-randomized observational studies where patients are selected from large datasets exist. With more than 131 million people, 66 percent of all adults in the United States, using prescription drugs and with utilization particularly high for vulnerable groups, including older adults and those with chronic conditions[1], the FDA must take the time to understand these issues before RWE can become a regular part of submissions and regulatory decision making.[2] A recent workshop hosted by the Duke-Margolis Health Policy Center brought together representatives from the FDA, industry, academia, and non-profit organizations to discuss the current and future activities of the FDA's RWE Program for drugs and biological products. Participants explored the opportunities and challenges of using RWD and RWE in regulatory decision-making and how to build on the accomplishments and knowledge gained in the 6 years since FDA published their Framework for FDA’s Real-World Evidence Program in 2018. FDA is actively seeking input to help facilitate the future direction of the RWE Program, see FDA Docket: FDA-2024-N-5057 for more information, the comment period ends 1/13/2025.
Center for Clinical Real-World Data Integration (CCRI)
This workshop covered a lot of ground and generated a lot of discussion and insights covered below. The big announcement at this meeting came when the third speaker, Dr. Corrigan-Curay, formerly CDER’s Office of Medical Policy Director and now the Principal Deputy Center Director, began her RWE Future Directions session with the announcement of the new Center for Clinical Real-World Data Integration (CCRI). The center has been created to assist with the increasing volume and complexity of regulatory submissions involving RWD and the use of emerging technologies such as artificial intelligence (AI) and digital health tools (DHTs) to collect, organize, and analyze RWD. CCRI will focus on the following:
· Scientific Review and Policy
· Coordinated Outreach and Engagement
· Research Initiatives
· Knowledge Management, including training
Agenda Highlights
Commissioner Califf opened the meeting remotely from California, where he self-reported tromping around pastures engaged in work on the avian influenza outbreak in cows. He expressed his enthusiasm for RWD and RWE, and its importance for the future of FDA and for therapeutics and diagnostics. Dr. John Concato, Associate Director for Real-World Evidence Analytics at CDER, took the stage next and provided an overview of FDA’s RWE Program followed by Dr. Corrigan-Curay’s insights on RWE’s future, including the announcement of the new CCRI. She was then joined on stage by Dr. Concato for a fireside chat. After a break, there were two panel sessions Opportunities and Challenges When Using Real-World Data, and the second panel and last session: Methodological Considerations for Real-World Evidence. Panelists in both sessions were from FDA, industry, academia, and nonprofits.
Overview of FDA’s RWE Program
John Concato, Associate Director for Real-World Evidence (RWE) Analytics in the Office of Medical Policy (OMP), began his session with a historical overview of RWD, starting in the 1990s.[3] He described how medical students and epidemiologists were traditionally taught using the "pyramid of strength of evidence," (see Fig. 1) which places randomized
controlled trials (RCTs) at the top and observational studies lower down, but not the lowest. This pyramid, though foundational, is an oversimplification, Dr. Concato noted, and that while RCTs are more trustworthy, the interpretation that non-randomized studies are untrustworthy is incorrect. This blog author also wants to point out that observational studies still rank higher than anecdotes and personal opinions, which are the weakest level of evidence. The origins of the pyramid date back to 1976 when The Canadian Task Force on the Periodic Health Examination was established to determine how the periodic health examination might enhance or protect the health of the population.[4] This may be the topic of a future blog post.!
Figure 1: Strength of Evidence Pyramid
Dr. Concato highlighted debates in peer-reviewed literature over the merits of real-world data (RWD), including the Hormone Replacement Therapy (HRT) study, where differences in analysis impacted coronary heart disease risk evaluations. His key takeaway: getting RWE right is challenging but achievable.
Two recent approvals based on RWE exemplify its potential. A new indication for Prograf was approved based solely on a non-interventional study, and another approval for a dose regimen change for Vimpat also used RWE data. Both applications met the FDA’s 21 CFR 314.126 requirement for “adequate and well-controlled studies” with validity comparable to RCTs. Dr. Concato noted, as he often does when discussing RWE, that FDA does not require him to memorize this regulation, he just repeats it so often that he has memorized it.
Prograf’s approval involved several factors that contributed to the successful use of RWE:
The source data was a registry covering all lung transplants during a specified period.
An observational design comparing treatment to historical controls, with an analysis plan and patient-level data provided to the FDA.
The outcomes used were organ rejection and death, where the dramatic effect of treatment minimized bias as an explanation.
Dr. Concato addressed challenges in using RWE, in the areas of source data fitness for use, study design and interpretation (e.g., confounding and immortal time bias), and proper study conduct, including lack of access to patient-level data. He emphasized that RWD and RWE are not new concepts; rather, electronic access to clinical data is evolving, making it more relevant and reliable. Next, he reviewed FDA’s extensive RWE-related accomplishments, such as guidance documents, demonstration projects on data, study design, and tools, and external collaborations, including those with ICH and EMA., noting that these efforts satisfy 21st Century Cures Act and PDUFA mandates. Dr. Concato concluded by citing a 2022 NEJM article he co-authored, reaffirming FDA’s commitment to robust policy development and maintaining evidentiary standards to protect and promote public health. The progress of the FDA’s RWE program underscores this dedication.
RWE Future Directions
Dr. Corrigan-Curay began her session with the announcement of the new Center for Clinical Real-World Data Integration (CCRI), as noted above. The Center will help CDER create a more centralized structure to help define strategic objectives, promote consistent scientific review and knowledge management, coordinate research goals, and engage the external real-world evidence (RWE) ecosystem. Its vision is to optimize the use of RWD when used to inform the effectiveness and safety of drugs and biological products, and its mission is to advance the evaluation of RWD by addressing gaps in regulatory science and policy related to RWE and by communicating CDER's priorities regarding using RWE in support of pre- and post-market regulatory decision-making in CDER.
Drs Concato and Dr. Corrigan-Curay answered questions during the Fireside Chat with FDA Leadership. John Concato mentioned his planned retirement, announced well before the election, he made sure to clarify. Dr. Corrigan-Curay did hint that he may be involved in a part-time capacity, but it remains to be seen if the 21 CFR 314.126 will keep its pole position in Dr. Concato’s memory or be replaced with information from other pursuits.
After a break, there were two panel sessions, which allowed panelists to give 5-minute opening remarks and then answer questions from the panel moderator and the audience. I have summarized the remarks and key points of discussion below.
Panel Session 1: Opportunities and Challenges When Using Real-World Data
Dr. Tala Fakhouri, the Associate Director for Data Science and Artificial Intelligence in the Office of Medical Policy at CDER, highlighted the transformative potential of artificial intelligence (AI) in accelerating drug development across all phases, from preclinical research to post-market surveillance. She emphasized that AI can be a powerful tool for generating evidence from RWD, but its success depends on the same prerequisites as RWD itself: high-quality, fit-for-purpose data. Dr. Fakhouri pointed out the need for “multi-data fluency” within organizations, urging greater collaboration between data scientists and domain experts—a challenge bioinformatics successfully tackled in the (now historic) 1990s, so maybe some lessons to be learned there.
Donna Rivera, the Associate Director for Pharmacoepidemiology in the Oncology Center of Excellence (OCE) at FDA, shared insights from the OCE, which has conducted over 200 consultations involving RWD applications. She emphasized the importance of ensuring data quality, particularly in pragmatic clinical trials, and stressed the methodological rigor required to enable RWD to yield insights that provide patient benefit. Dr. Rivera also underscored the work being done to Understand how RWD can be applied for regulatory purposes and the potential for RWD to modernize evidence generation across the agency.
Jingyu (Julia) Luan, Executive Regulatory Science Director at AstraZeneca (AZ), emphasized that data quality is the bedrock of their work. She referenced AZ’s DAPA-MI trial, which demonstrated the feasibility of registry-based randomized controlled trials (R-RCTs). This innovative approach accelerated trial recruitment, reduced costs, and minimized patient site visits, showcasing the efficiency gains possible with high-quality RWD.
Nicole Mahoney, an Executive Director for Regulatory Policy at Novartis, discussed creating workflows to ensure data reliability and relevance within the industry. She raised the practical challenges of assessing vendors and measuring data quality for regulatory purposes. She also touched on the complexities of harmonizing definitions and guidance across international regulatory bodies, a task that likely falls outside the FDA’s remit but could be addressed by collaborative initiatives, perhaps a role for TransCelerate?
Dan Riskin, MD, MBA, Founder and CEO of Verantos and a Clinical Professor at Stanford University, pointed out the potential safety implications of using electronic health record (EHR) data in learning health systems if the underlying data is not accurate. Drawing on findings from the TRUST study, his findings show that data quality can significantly influence patient cohort definitions and study outcomes. This blog author’s takeaway: there is a good reason that trials have historically been done outside the healthcare system. Riskin advocated for further research into the implications of data quality.
Themes Explored and Key Takeaways
Data Quality: Panelists universally agreed on the importance of reliable, relevant, and high-quality data. However, there is no consensus yet on how to systematically improve RWD quality or establish robust reporting standards for reporting on data quality.
Role of AI: AI’s potential to enhance RWD analysis was a central topic. Discussions included creating datasets for AI testing, ensuring algorithm generalizability, and addressing the dependence of generalizability on data relevance.
Regulatory Considerations: There are highly varied capabilities in creating datasets; some extract data well, including verification and traceability, but others don’t. The panel explored how to maximize RWD usefulness while maintaining scientific rigor.
Context Dependence: There are no hard and fast rules for RWD, it depends a lot on the question or questions being asked of the data. What to measure in the data to prove that it is fit for use and of high enough quality for regulatory purposes is use-dependent.
Collaborative Approaches: Bridging the gap between data scientists, engineers, and domain experts is essential for the effective use of RWD and creating the right analysis tools.
Registries as a Solution: Patient registries could play a pivotal role in improving data availability and reliability, as seen in the development of AstraZeneca’s DAPA-MI trial. (This blog author agrees and recommends interested readers investigate the role of a registry in developing treatments for cystic fibrosis.)
Methodological Considerations for Real-World Evidence
Jennie Li, PhD and Associate Director for RWE at the Office of Surveillance and Epidemiology in CDER, highlighted four pivotal efforts at the FDA to advance RWE methodologies: New tools and methods for pulling data from EHRs and the use of data to pull more reliable cohorts, the Use of Sentinel to estimate the potential impact of confounding and bias in RWD, a framework and tools to identify patterns of missingness and work on a causal inference framework. Ms. Li emphasized key questions researchers should address when designing RWE studies, such as selecting appropriate endpoints, assessment frequency, what comparators can be used if the standard of care is not measurable and understanding disease progression and known pragmatic factors.
Yun Lu, PhD and Deputy Division Director at the Division of Analytics and Benefit-Risk Assessment in CBER’s of Biostatistics and Pharmacovigilance, provided insights into the unique challenges of vaccine research in RWD. Vaccines are administered to healthy populations and may introduce biases as vaccine recipients may inherently be healthier than people who chose not to be vaccinated. To address these biases, the FDA is developing statistical methods to reduce unmeasured confounding in vaccine effectiveness studies.
Simon Dagenais, PhD and Lead for Methods & Innovation in the RWE Platform at Pfizer, advocated for hybrid RCT/RWE approaches in chronic disease research, such as Type 2 Diabetes (T2D) where the disease and endpoints are well characterized and where pre-market approval studies often require 8-10K patients, take over 6 years, and cost a lot of money. His suggestions included utilizing augmented controls derived from RWD and implementing a mix of passive follow-up and scheduled visits to enhance study efficiency.
Stefan James, Professor of Cardiology and Scientific Director of Uppsala Clinical Research Center at Uppsala University, highlighted the role national registries can play. He suggested national registries with defined core variables and federated data access could facilitate analyses. He also noted that complex study outcomes would likely need some level of adjudication.
Sebastian Schneeweiss, M.D., Sc.D., and Professor of Medicine at Harvard Medical School (among other positions), underscored the need for rigorous benchmarking and data quality assessment, a theme heard in session 1. He built on Dr. Dagenais’s use case and described a possible way to expand effectiveness working with RWD, including first benchmarking a drug’s early RWD findings against the phase 3 trial results to validate data reliability and then expanding from there, stepwise, into other populations and/or clinical endpoints. He also addressed data quality, noting that many would be surprised at how poor their data quality is if they did measure it. However, understanding the flaws is a necessary step for improving the quality.
Themes Explored and Key Takeaways
Endpoints and Validation: While overall survival (OS) remains a straightforward endpoint, more complex endpoints require validation and/or adjudication and need to consider the disease trajectory, treatment landscape, and study context.
Data Quality and Reporting: Robust RWE requires high-quality data. Researchers should focus on describing datasets, defining variables, and setting quality benchmarks to enhance validity and reproducibility.
Understanding measurement imperfections and aligning data sources with desired characteristics are important for ensuring study results are meaningful.
Sharing Lessons Learned: Given the complexity and cost of validating RWD, panelists emphasized the importance of sharing best practices and lessons learned. Not every sponsor should have to analyze a dataset to understand the quality, data vendors could provide this information.
Frameworks and Methodologies: Sentinel’s advancements in handling missing data, ongoing work to understand causal inference, and exploration of confounding and bias in linked versus unlinked datasets will provide valuable tools for addressing common RWE challenges.
Pragmatic Approaches/ Innovative Data Integration: Combining structured and unstructured data, using the former for cohort selection and the latter for subsequent characterization of other factors.
Methods blending RWD with traditional RCT data to create adaptable and efficient study designs.
Conclusion
Real-world evidence has a pivotal role in the future of regulatory science. The FDA is continuing its commitment to supporting the use of RWE in both premarket approvals and post-market surveillance through research, demonstration projects, and initiatives like CCRI. There are many opportunities for innovation with RWD, RWE, and AI that can both help to improve data quality and analyze data. However, fitness for use, data quality metrics, and appropriate analysis methods are still evolving and need to be better understood before the information can be used without close human supervision and analyses. The FDA has not changed its data standards[5] and will continue to require “[a]dequate and well-controlled studies” for regulatory decision-making (see 21 CFR 314.126) whether the data comes from randomized clinical trials, observational studies, pragmatic trials, or another design.
Progress in RWE requires a unified effort from all stakeholders: regulators, researchers, industry, physicians, patients, payers, and EHR vendors. The FDA is interested in hearing from all stakeholders about their experience with RWD and RWE and how the FDA can best move forward to use this data to improve the health of the public. Please comment on FDA Docket: FDA-2024-N-5057.
A big thanks to Duke Margolis, FDA, the speakers, organizers, and participants for an interesting and thought-provoking meeting. I look forward to updates from the CCRI and continued progress on the use of RWD and RWE to help patients live healthier lives.
Footnotes
[1] https://hpi.georgetown.edu/rxdrugs/
[2] Note that RWD has long been used by sponsors to inform study design. Also, FDA has used submitted RWE in regulatory decision-making.
[3] Yes, it is hard to swallow but the 1990s is now considered historic.
[4] Canadian Task Force on the Periodic Health Examination (3 November 1979). "Task Force Report: The periodic health examination". Can Med Assoc J. 121 (9): 1193–1254. PMC 1704686. PMID 115569
[5] In this context, I am using standards to mean “quality” or “grade”, not “the rules that define how data is structured, described, and shared”. It is worth noting that FDA has not changed their data standards for submission of RWE, CDISC SDTM and ADaM are still required, but they are actively exploring whether a change is needed .