Machine Learning in Pharmacovigilance
Highlights from a Panel Discussion

Shaun Comfort
Genentech, A Member of the Roche Group

Elenee Argentinis
IBM Watson Health, Life Sciences Solutions

Jennifer Fine
Genentech, A Member of the Roche Group

Bruce Donzanti
Genentech, A Member of the Roche Group

L

ike many other healthcare areas, the field of pharmacovigilance (PV) is dealing with ever increasing volumes of data. At a panel discussion during the 2018 DIA Pharmacovigilance and Risk Management Strategies Conference in Washington, DC, members from Genentech/Roche, IBM, the US FDA, and the UK Medicines and Healthcare Products Regulatory Agency (MHRA) discussed practical considerations for applying novel technologies like cognitive computing and machine learning (ML) to support pharmacovigilance.

The panel provided a variety of perspectives ranging from the pharmaceutical industry and regulatory authorities to cognitive IT. Here are some key takeaways:

  • Accurately measuring the human “gold standard” performance in common PV tasks is critical, allowing us to set benchmarks for evaluating the effectiveness and accuracy of new technologies.
  • A “human-technology partnership” between human PV experts and cognitive technology may simultaneously offer speed, scale, consistency, and data quality improvements, while maintaining human oversight and control at key decision points.
  • Thoughtfully training cognitive and AI solutions like ML is critical to their performance. Companies will need to apply their best “business knowledge experts” to reviewing and identifying training data that are representative, diverse, consistent, accurate, and complete.
  • Both FDA and MHRA expressed interest in exploring use of these technologies in PV, recognizing that they are actively explored by industry.
  • To demonstrate compliance, new technologies will need to fit within current regulatory and legal frameworks and be equipped with robust quality systems. For audits and inspections, industry will need to be transparent about how such systems are trained and their performance is evaluated.
  • The panel recognized that the great potential of ML and cognitive computing in PV might best be achieved in partnership with the pharmaceutical industry, IT, and regulatory authorities.
  • Read more to review the current state of PV and the major themes discussed during the session.

Current State

Much of the effort in PV is focused on identifying, collecting, evaluating, and transforming relevant data into usable information for regulators and companies to address safety concerns and inform the public and prescribers. Currently, the industry addresses increasing data volumes by adding new safety team members and utilizing various internal management models. But there are limits to how much growth can be sustained without enlisting outsourcing services to keep up with the escalating data challenge.

An important challenge beyond volume and logistics lies in the performance limitations of human beings tasked with processing large volumes of information, affecting both accuracy and consistency of data interpretation (commonly referred to as interrater agreement). As teams grow and expand across geographies, these limitations are further exacerbated.

Looking to the Future

Early innovators in the pharmaceutical industry, alongside health authorities and technology providers, have begun piloting technologies like cognitive computing and ML as an approach to achieve AI to help address data volume challenges. These early experiences can provide us with cautionary insights and important questions that ought to be considered as AI and cognitive computing are entering production systems and are utilized at scale.

Setting Performance Expectations: The Human Benchmark

PV is held to high regulatory standards of quality, accuracy, and timeliness. As new technologies enter the PV space, performance expectations and measures will influence how the pharmaceutical industry and the safety ecosystem will use and monitor them.

Using human performance in PV tasks as a standard may be the appropriate starting point as it reflects the current state of PV and is regarded as the default “ground truth.” However, research has shown poor to moderate agreement among trained experts reviewing the same information.

Accurately measuring the current human “gold standard” performance in common PV tasks therefore is a critical first step, allowing us to set benchmarks for evaluating the effectiveness of novel technologies.

The Human-Technology Partnership

A “human-technology partnership” between PV experts and cognitive technology may provide a potential solution in that it combines the best of both worlds, offering improvements in speed, scale, consistency, and data quality.

If new technologies performed the highly repetitive manual labor of extracting and reading incoming reports, it would free up time for PV safety teams to review and revise the information in reports, confirming that the reports are complete, accurate, and of high quality. This would allow for quicker and more consistent safety actions than the current manual approach.

The Importance of the Ground Truth

Training cognitive and AI solutions to increase their performance reliability calls for a consistent, high-quality “ground truth” that is representative of the entities and classifications in production. Companies will need to use their best “business knowledge experts” to review and identify training data that are representative, diverse, consistent, accurate, and complete:

  • Representativeness: Training data should be representative enough to cover most of the typical safety cases processed, being as similar as possible to the final production data.
  • Diversity: Training data should include as many variations of the target attributes as possible, covering variations in drug descriptions, adverse events, report types, and seriousness.
  • Consistency: Training data should be consistent across the corpus. For example, training data that is used to train an ICSR model must contain consistent representations of key elements, including patient, reporter, drug, and event.
  • Accuracy: Training data should be as accurate as possible. For example, training data containing frequent incorrect classifications of valid and invalid ICSRs will produce biased classification models that perform poorly.
  • Completeness: Training data should contain all the information needed for the desired task. Systems trained with training data that are missing key elements to make decisions on ICSR seriousness, for example, will perform poorly.

Regulatory Perspectives

Pharmacovigilance is embedded in a global regulatory context. Representatives from the FDA and MHRA offered their perspectives and showed interest in these new technologies, recognizing that they are actively explored by industry. FDA has been piloting work with AI in a few areas for several years. This includes pilots with the goal of using AI to identify Individual Case Safety Reports in the FDA’s Adverse Event Reporting System that are more likely to contain information relevant to assessing the causal relationship between a drug and an adverse event. That said, both agencies predicted that it probably will take several years before the industry will see broad adoption of AI.

Incorporating Compliance and Quality Management Systems

MHRA emphasized that future AI systems will need to fit within current regulatory and legal frameworks and demonstrate compliance with robust quality management systems. This may require new skill sets for both industry and regulatory authorities. Early adopters will need to build robust quality systems around this new technology to satisfy compliance needs. And industry will need to be transparent about how such systems are trained, how performance is defined and measured, and how system decisions are made during audits and inspections.

Conclusion

This is an exciting time with a promising technology. The panel recognized the great potential of ML as an initial approach to achieve AI and cognitive computing in PV but admitted that much work remains to be done in this space before we will see the first fully implemented ML/Cognitive PV system. The panel did suggest that the opportunities may best be achieved in partnership with the pharmaceutical industry, IT, and regulatory authorities to help lay the framework for how to advance this technology to benefit public health while maintaining appropriate oversight and management.