Validation of the AI/RHEUM knowledge base with data from consecutive rheumatological outpatients

Methods Inf Med. 1992 Sep;31(3):175-81.


The diagnostic accuracy of AI/RHEUM, an experimental expert system for support in the diagnosis of rheumatic diseases, was assessed using a collection of data in a cohort of 1,570 consecutive outpatients of a Dutch rheumatological clinic. Computer diagnoses based on these data and diagnostic predictions made by rheumatologists were compared with reference diagnoses that had been obtained by consensus of rheumatologists after 6-12 months follow-up. Performance of the tested version of the AI/RHEUM knowledge base is presented by various methods. Sensitivity varied between 29% and 100% for different rheumatological diseases. Average sensitivity and specificity for all 26 diagnoses present in the knowledge base were 67% and 98%, respectively. Performance according to the level of confidence indicated that 78% of the "definite", 65% of the "probable", and 33% of the "possible" conclusions made by AI/RHEUM were in agreement with the reference diagnoses. These results approximated the predictions made by rheumatologists after a single, initial examination. The system was less accurate than it had appeared in previous evaluation studies with complex clinical cases. The AI/RHEUM knowledge base needed refining to diagnose early rheumatic complaints. This study further illustrates the need for objective and informative parameters for expressing accuracy of diagnostic support systems.

MeSH terms

  • Adolescent
  • Adult
  • Aged
  • Aged, 80 and over
  • Artificial Intelligence*
  • Child
  • Child, Preschool
  • Diagnosis, Computer-Assisted*
  • Expert Systems
  • Female
  • Humans
  • Male
  • Middle Aged
  • Referral and Consultation
  • Rheumatic Diseases / diagnosis
  • Rheumatology / methods*
  • Sensitivity and Specificity