Criteria2Query: a natural language interface to clinical databases for cohort definition

Chi Yuan; Patrick B Ryan; Casey Ta; Yixuan Guo; Ziran Li; Jill Hardin; Rupa Makadia; Peng Jin; Ning Shang; Tian Kang; Chunhua Weng

doi:10.1093/jamia/ocy178

Criteria2Query: a natural language interface to clinical databases for cohort definition

J Am Med Inform Assoc. 2019 Apr 1;26(4):294-305. doi: 10.1093/jamia/ocy178.

Authors

Chi Yuan^{1

2}, Patrick B Ryan^{1

3}, Casey Ta¹, Yixuan Guo¹, Ziran Li¹, Jill Hardin³, Rupa Makadia³, Peng Jin¹, Ning Shang¹, Tian Kang¹, Chunhua Weng¹

Affiliations

¹ Department of Biomedical Informatics, Columbia University, New York, New York, USA.
² Department of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing, Jiangsu Province, P.R. China.
³ Epidemiology Analytics, Janssen Research and Development, Titusville, New Jersey, USA.

Abstract

Objective: Cohort definition is a bottleneck for conducting clinical research and depends on subjective decisions by domain experts. Data-driven cohort definition is appealing but requires substantial knowledge of terminologies and clinical data models. Criteria2Query is a natural language interface that facilitates human-computer collaboration for cohort definition and execution using clinical databases.

Materials and methods: Criteria2Query uses a hybrid information extraction pipeline combining machine learning and rule-based methods to systematically parse eligibility criteria text, transforms it first into a structured criteria representation and next into sharable and executable clinical data queries represented as SQL queries conforming to the OMOP Common Data Model. Users can interactively review, refine, and execute queries in the ATLAS web application. To test effectiveness, we evaluated 125 criteria across different disease domains from ClinicalTrials.gov and 52 user-entered criteria. We evaluated F1 score and accuracy against 2 domain experts and calculated the average computation time for fully automated query formulation. We conducted an anonymous survey evaluating usability.

Results: Criteria2Query achieved 0.795 and 0.805 F1 score for entity recognition and relation extraction, respectively. Accuracies for negation detection, logic detection, entity normalization, and attribute normalization were 0.984, 0.864, 0.514 and 0.793, respectively. Fully automatic query formulation took 1.22 seconds/criterion. More than 80% (11+ of 13) of users would use Criteria2Query in their future cohort definition tasks.

Conclusions: We contribute a novel natural language interface to clinical databases. It is open source and supports fully automated and interactive modes for autonomous data-driven cohort definition by researchers with minimal human effort. We demonstrate its promising user friendliness and usability.

Keywords: cohort definition; common data model; natural language interfaces to database; natural language processing.

Publication types

Research Support, N.I.H., Extramural
Research Support, Non-U.S. Gov't

MeSH terms

Clinical Trials as Topic*
Databases as Topic*
Eligibility Determination / methods*
Humans
Information Storage and Retrieval / methods*
Machine Learning
Natural Language Processing*
Patient Selection*
User-Computer Interface*

Abstract

Publication types

MeSH terms

Grants and funding