Identifying Biomedical Entities for Datasets in Scientific Articles: 4-Step Cache-Augmented Generation Approach Using GPT-4o and PubTator 3.0

Claudia Giuliani; Gita Benadi; Felix Engel; Jonas Werner; Manuel Watter; Guido Schwarzer; Olaf Groß; Robert Zeiser; Harald Binder; Klaus Kaier

doi:10.2196/73822

Identifying Biomedical Entities for Datasets in Scientific Articles: 4-Step Cache-Augmented Generation Approach Using GPT-4o and PubTator 3.0

JMIR Form Res. 2025 Nov 20:9:e73822. doi: 10.2196/73822.

Authors

Claudia Giuliani¹, Gita Benadi¹, Felix Engel¹, Jonas Werner¹, Manuel Watter¹, Guido Schwarzer¹, Olaf Groß^{2

3}, Robert Zeiser^{3

4}, Harald Binder^{1

3}, Klaus Kaier^{1

3}

Affiliations

¹ Institute of Medical Biometry and Statistics, Medical Faculty and Medical Center, University of Freiburg, Stefan-Meier-Str. 26, Freiburg, 79104, Germany, 49 076127083739.
² Institute of Neuropathology, Medical Faculty and Medical Center, University of Freiburg, Freiburg, Germany.
³ Center for Integrative Biological Signaling Studies, University of Freiburg, Freiburg, Germany.
⁴ Department of Medicine I, Medical Faculty and Medical Center, University of Freiburg, Freiburg, Germany.

PMID: 41264807
PMCID: PMC12633840
DOI: 10.2196/73822

Abstract

Background: The accurate extraction of biomedical entities in scientific articles is essential for effective metadata annotation of research datasets, ensuring data findability, accessibility, interoperability, and reusability in collaborative research.

Objective: This study aimed to introduce a novel 4-step cache-augmented generation approach to identify biomedical entities for an automated metadata annotation of datasets, leveraging GPT-4o and PubTator 3.0.

Methods: The method integrates four steps: (1) generation of candidate entities using GPT-4o, (2) validation via PubTator 3.0, (3) term extraction based on a metadata schema developed for the specific research area, and (4) a combined evaluation of PubTator-validated and schema-related terms. Applied to 23 articles published in the context of the Collaborative Research Center OncoEscape, the process was validated through supervised, face-to-face interviews with article authors, allowing an assessment of annotation precision using random-effects meta-analysis.

Results: The approach yielded a mean of 19.6 schema-related and 6.7 PubTator-validated biomedical entities per article. Within the study's specific context, the overall annotation precision was 98% (95% CI 94%-100%), with most prediction errors concentrated in articles outside the primary basic research domain of the schema. In a subsample (n=20), available supplemental material was included in the prediction process, but it did not improve precision (98%, 95% CI 95%-100%). Moreover, the mean number of schema-related entities was 20.1 (P=.56) and the mean number of PubTator-validated entities was 6.7 (P=.68); these values did not increase with the additional information provided in the supplement.

Conclusions: This study highlights the potential of large language model-supported metadata annotation. The findings underscore the practical feasibility of full-text analysis and suggest its potential for integration into routine workflows for biomedical metadata generation.

Keywords: AI; CAG; GPT-4o; PubTator 3.0; artificial intelligence; biomedical entities; cache-augmented generation; metadata annotation.

© Claudia Giuliani, Gita Benadi, Felix Engel, Jonas Werner, Manuel Watter, Guido Schwarzer, Olaf Groß, Robert Zeiser, Harald Binder, Klaus Kaier. Originally published in JMIR Formative Research (https://formative.jmir.org).

MeSH terms

Biomedical Research*
Data Mining* / methods
Datasets as Topic*
Humans
Information Storage and Retrieval* / methods
Metadata*