How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective

Marvin Kopka; Markus A Feufel; Eta S Berner; Malte L Schmieding

doi:10.1177/20552076231194929

How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective

Digit Health. 2023 Aug 21:9:20552076231194929. doi: 10.1177/20552076231194929. eCollection 2023 Jan-Dec.

Authors

Marvin Kopka^{1

2}, Markus A Feufel¹, Eta S Berner³, Malte L Schmieding²

Affiliations

¹ Department of Psychology and Ergonomics (IPA), Division of Ergonomics, Technische Universität Berlin, Berlin, Germany.
² Institute of Medical Informatics, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, Germany.
³ Department of Health Services Administration, University of Alabama at Birmingham, Birmingham, AL, USA.

Abstract

Objective: To evaluate the ability of case vignettes to assess the performance of symptom checker applications and to suggest refinements to the methodology used in case vignette-based audit studies.

Methods: We re-analyzed the publicly available data of two prominent case vignette-based symptom checker audit studies by calculating common metrics of test theory. Furthermore, we developed a new metric, the Capability Comparison Score (CCS), which compares symptom checker capability while controlling for the difficulty of the set of cases each symptom checker evaluated. We then scrutinized whether applying test theory and the CCS altered the performance ranking of the investigated symptom checkers.

Results: In both studies, most symptom checkers changed their rank order when adjusting the triage capability for item difficulty (ID) with the CCS. The previously reported triage accuracies commonly overestimated the capability of symptom checkers because they did not account for the fact that symptom checkers tend to selectively appraise easier cases (i.e., with high ID values). Also, many case vignettes in both studies showed insufficient (very low and even negative) values of item-total correlation (ITC), suggesting that individual items or the composition of item sets are of low quality.

Conclusions: A test-theoretic perspective helps identify previously undetected threats to the validity of case vignette-based symptom checker assessments and provides guidance and specific metrics to improve the quality of case vignettes, in particular by controlling for the difficulty of the vignettes an app was (not) able to evaluate correctly. Such measures might prove more meaningful than accuracy alone for the competitive assessment of symptom checkers. Our approach helps elaborate and standardize the methodology used for appraising symptom checker capability, which, ultimately, may yield more reliable results.

Keywords: Digital health; care navigation; case vignettes; methodology; patient-centered care; self-triage; symptom checker; test theory; urgency assessment.