Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications

J Am Med Inform Assoc. 2012 Sep-Oct;19(5):713-8. doi: 10.1136/amiajnl-2012-000852. Epub 2012 May 12.


Objective: We describe a novel, crowdsourcing method for generating a knowledge base of problem-medication pairs that takes advantage of manually asserted links between medications and problems.

Methods: Through iterative review, we developed metrics to estimate the appropriateness of manually entered problem-medication links for inclusion in a knowledge base that can be used to infer previously unasserted links between problems and medications.

Results: Clinicians manually linked 231,223 medications (55.30% of prescribed medications) to problems within the electronic health record, generating 41,203 distinct problem-medication pairs, although not all were accurate. We developed methods to evaluate the accuracy of the pairs, and after limiting the pairs to those meeting an estimated 95% appropriateness threshold, 11,166 pairs remained. The pairs in the knowledge base accounted for 183,127 total links asserted (76.47% of all links). Retrospective application of the knowledge base linked 68,316 medications not previously linked by a clinician to an indicated problem (36.53% of unlinked medications). Expert review of the combined knowledge base, including inferred and manually linked problem-medication pairs, found a sensitivity of 65.8% and a specificity of 97.9%.

Conclusion: Crowdsourcing is an effective, inexpensive method for generating a knowledge base of problem-medication pairs that is automatically mapped to local terminologies, up-to-date, and reflective of local prescribing practices and trends.

Publication types

  • Research Support, N.I.H., Extramural
  • Research Support, U.S. Gov't, Non-P.H.S.

MeSH terms

  • Adolescent
  • Adult
  • Child
  • Crowdsourcing*
  • Drug Therapy, Computer-Assisted*
  • Electronic Health Records
  • Humans
  • Knowledge Bases*
  • Medical Records, Problem-Oriented*
  • Retrospective Studies
  • Sensitivity and Specificity
  • Texas