The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia

Abstract

Objectives: To investigate the quality of diagnostic and triage advice provided by free website and mobile application symptom checkers (SCs) accessible in Australia.

Design: 36 SCs providing medical diagnosis or triage advice were tested with 48 medical condition vignettes (1170 diagnosis vignette tests, 688 triage vignette tests).

Main outcome measures: Correct diagnosis advice (provided in first, the top three or top ten diagnosis results); correct triage advice (appropriate triage category recommended).

Results: The 27 diagnostic SCs listed the correct diagnosis first in 421 of 1170 SC vignette tests (36%; 95% CI, 31-42%), among the top three results in 606 tests (52%; 95% CI, 47-59%), and among the top ten results in 681 tests (58%; 95% CI, 53-65%). SCs using artificial intelligence algorithms listed the correct diagnosis first in 46% of tests (95% CI, 40-57%), compared with 32% (95% CI, 26-38%) for other SCs. The mean rate of first correct results for individual SCs ranged between 12% and 61%. The 19 triage SCs provided correct advice for 338 of 688 vignette tests (49%; 95% CI, 44-54%). Appropriate triage advice was more frequent for emergency care (63%; 95% CI, 52-71%) and urgent care vignette tests (56%; 95% CI, 52-75%) than for non-urgent care (30%; 95% CI, 11-39%) and self-care tests (40%; 95% CI, 26-49%).

Conclusion: The quality of diagnostic advice varied between SCs, and triage advice was generally risk-averse, often recommending more urgent care than appropriate.

Keywords: Diagnosis; Internet; Mobile applications; Signs and symptoms; eHealth.