Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2010 Dec;43(6):962-71.
doi: 10.1016/j.jbi.2010.07.007. Epub 2010 Jul 27.

Automatically Extracting Information Needs From Complex Clinical Questions

Affiliations
Free PMC article

Automatically Extracting Information Needs From Complex Clinical Questions

Yong-gang Cao et al. J Biomed Inform. .
Free PMC article

Abstract

Objective: Clinicians pose complex clinical questions when seeing patients, and identifying the answers to those questions in a timely manner helps improve the quality of patient care. We report here on two natural language processing models, namely, automatic topic assignment and keyword identification, that together automatically and effectively extract information needs from ad hoc clinical questions. Our study is motivated in the context of developing the larger clinical question answering system AskHERMES (Help clinicians to Extract and aRrticulate Multimedia information for answering clinical quEstionS).

Design and measurements: We developed supervised machine-learning systems to automatically assign predefined general categories (e.g. etiology, procedure, and diagnosis) to a question. We also explored both supervised and unsupervised systems to automatically identify keywords that capture the main content of the question.

Results: We evaluated our systems on 4654 annotated clinical questions that were collected in practice. We achieved an F1 score of 76.0% for the task of general topic classification and 58.0% for keyword extraction. Our systems have been implemented into the larger question answering system AskHERMES. Our error analyses suggested that inconsistent annotation in our training data have hurt both question analysis tasks.

Conclusion: Our systems, available at http://www.askhermes.org, can automatically extract information needs from both short (the number of word tokens <20) and long questions (the number of word tokens >20), and from both well-structured and ill-formed questions. We speculate that the performance of general topic classification and keyword extraction can be further improved if consistently annotated data are made available.

Figures

Figure 1
Figure 1
AskHERMES' system architecture. AskHERMES takes as input a question posed by a clinician. Question Analysis automatically extracts information needs. Document Retrieval retrieves relevant documents (MEDLINE and WWW). Answer Extraction automatically identifies the sentences that provide answers to questions. Summarization condenses the text by removing the redundant sentences and by generating a coherent summary. Answer Presentation presents the summary to the user who posed the question.
Figure 2
Figure 2
Three types of indicative keywords and examples
Figure 3
Figure 3
The classification performance of topic assignment and the corresponding training size (training size =10*Ln (Number of Training Questions)).
Figure 4
Figure 4
(a) Number of questions as a function of number of categories assigned to questions of the ClinicalQuestions Collection; (b) Classification performance of topic assignment as a function of number of categories assigned to a question
Figure 5
Figure 5
(a) Recall, precision, and F1 score of keyword extraction (using CRFs) as a function of question length. (b) Number of questions as a function of question length.

Similar articles

See all similar articles

Cited by 10 articles

See all "Cited by" articles

Publication types

MeSH terms

LinkOut - more resources

Feedback