rtificial intelligence, particularly large language models utilizing custom prompts, can reduce the manual effort of scientific literature human review teams.

Detecting adverse events (AEs) in scientific literature is a key pharmacovigilance obligation for a Market Authorization Holder (MAH). Pharmacovigilance teams review scientific literature for the most part manually in order to determine whether sufficient information is contained for a reported event to be classified as a valid Individual Case Safety Report or other human safety information of significance.

Currently, technology exists to help prioritize results from rule-based scientific literature searches and alerts. Human reviewers then conduct manual assessments to determine 1) if an abstract contains reportable adverse event information, 2) if an abstract contains pregnancy information, and 3) if the full-text article is an article of interest for signal detection.

This article provides results of an assessment of whether a commercial large language model (LLM) and custom prompts could automatically answer these questions and appropriately summarize abstracts and full-text articles for later human review.

How We Conducted Our Assessment

We collected 89 publicly available abstracts and their corresponding full text that were previously manually assessed by a pharmacovigilance scientific literature specialist. Twenty-five of these abstracts contained an AE, seven full-text articles contained pregnancy information, and 29 full-text articles contained potential safety signal information.

We developed custom prompts to answer these five questions:

1. Does the abstract contain a reportable AE?
2. Does the abstract contain pregnancy information?
3. Can a summarization be created that contains all safety-relevant and -pertinent clinical information from the abstract?
4. Can a summarization be created that contains all safety-relevant and -pertinent clinical information from the full-text article?
5. Does the full-text article contain a potential safety signal?

We then developed software code that provided each abstract and full-text article along with the relevant prompt to a commercially available LLM. We collected and reviewed the LLM’s responses for accuracy and completeness.

The system did not retain any memory or linkage between abstracts, full-text articles, or LLM prompts. We examined text only; we did not use optical character recognition to process images contained in the abstracts or full-text articles.

Our Assessment’s Outcome

Identification of reportable adverse events, pregnancy information, and safety signals was 100% accurate. Abstract summarization achieved 100% accuracy, while full-text summarization reached 94% accuracy (84/89). Challenges with text extraction from complex table structures within the full-text articles caused the remaining 6% of errors. Compared to an estimated manual review time of 890 minutes (~10 minutes/article), the LLM processed the batch (all abstracts and full-text articles across the five prompts) in 30 minutes, representing a theoretical ~97% reduction in processing time.

Discussion and Direction

Using a commercial LLM and custom prompts successfully achieved: 1) automatic analysis of abstracts for AE information; 2) automatic analysis of abstracts for pregnancy information; and 3) automatic analysis of full-text articles for interest in signal detection. We also demonstrated that an LLM with custom prompts could summarize abstracts and full-text articles.

Future work should improve the summarization capabilities while adhering to principles of trustworthy AI, including transparency, robustness, and appropriate human oversight, to ensure safe and responsible use of AI in pharmacovigilance workflows. Our work suggests a significant opportunity for future efficiency gains to enable the reduction of 100% human review of scientific literature over time.