Data extraction methods for systematic review (semi)automation: Update of a living systematic review

Lena Schmidt; Ailbhe N Finnerty Mutlu; Rebecca Elmore; Babatunde K Olorisade; James Thomas; Julian P T Higgins

doi:10.12688/f1000research.51117.2

Data extraction methods for systematic review (semi)automation: Update of a living systematic review

F1000Res. 2021 May 19:10:401. doi: 10.12688/f1000research.51117.2. eCollection 2021.

Authors

Lena Schmidt^{1

2

3}, Ailbhe N Finnerty Mutlu⁴, Rebecca Elmore², Babatunde K Olorisade^{3

5

6}, James Thomas⁴, Julian P T Higgins³

Affiliations

¹ NIHR Innovation Observatory, Newcastle University, Newcastle upon Tyne, NE4 5TG, UK.
² Sciome LLC, Research Triangle Park, North Carolina, 27713, USA.
³ Bristol Medical School, University of Bristol, Bristol, BS8 2PS, UK.
⁴ UCL Social Research Institute, University College London, London, WC1H 0AL, UK.
⁵ Evaluate Ltd, London, SE1 2RE, UK.
⁶ Cardiff School of Technologies, Cardiff Metropolitan University, Cardiff, CF5 2YB, UK.

Abstract

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search PubMed, ACL Anthology, arXiv, OpenAlex via EPPI-Reviewer, and the dblp computer science bibliography. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This living review update includes publications up to December 2022 and OpenAlex content up to March 2023. Results: 76 publications are included in this review. Of these, 64 (84%) of the publications addressed extraction of data from abstracts, while 19 (25%) used full texts. A total of 71 (93%) publications developed classifiers for randomised controlled trials. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. Data are available from 25 (33%), and code from 30 (39%) publications. Six (8%) implemented publicly available tools Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of literature review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. Between review updates, trends for sharing data and code increased strongly: in the base-review, data and code were available for 13 and 19% respectively, these numbers increased to 78 and 87% within the 23 new publications. Compared with the base-review, we observed another research trend, away from straightforward data extraction and towards additionally extracting relations between entities or automatic text summarisation. With this living review we aim to review the literature continually.

Keywords: Data Extraction; Natural Language Processing; Reproducibility; Systematic Reviews; Text Mining.

Publication types

Systematic Review

Grants and funding

We acknowledge funding from NIHR (LAM through NIHR Doctoral Research Fellowship (DRF-2018-11-ST2-048), and LS through NIHR Systematic Reviews Fellowship (RM-SR-2017-09-028)). LAM is a member of the MRC Integrative Epidemiology Unit at the University of Bristol. The views expressed in this article are those of the authors and do not necessarily represent those of the NHS, the NIHR, MRC, or the Department of Health and Social Care.