Multireader evaluation of radiologist performance for COVID-19 detection on emergency department chest radiographs

Clin Imaging. 2022 Feb;82:77-82. doi: 10.1016/j.clinimag.2021.10.018. Epub 2021 Nov 12.


Background: Chest radiographs (CXR) are frequently used as a screening tool for patients with suspected COVID-19 infection pending reverse transcriptase polymerase chain reaction (RT-PCR) results, despite recommendations against this. We evaluated radiologist performance for COVID-19 diagnosis on CXR at the time of patient presentation in the Emergency Department (ED).

Materials and methods: We extracted RT-PCR results, clinical history, and CXRs of all patients from a single institution between March and June 2020. 984 RT-PCR positive and 1043 RT-PCR negative radiographs were reviewed by 10 emergency radiologists from 4 academic centers. 100 cases were read by all radiologists and 1927 cases by 2 radiologists. Each radiologist chose the single best label per case: Normal, COVID-19, Other - Infectious, Other - Noninfectious, Non-diagnostic, and Endotracheal Tube. Cases labeled with endotracheal tube (246) or non-diagnostic (54) were excluded. Remaining cases were analyzed for label distribution, clinical history, and inter-reader agreement.

Results: 1727 radiographs (732 RT-PCR positive, 995 RT-PCR negative) were included from 1594 patients (51.2% male, 48.8% female, age 59 ± 19 years). For 89 cases read by all readers, there was poor agreement for RT-PCR positive (Fleiss Score 0.36) and negative (Fleiss Score 0.46) exams. Agreement between two readers on 1638 cases was 54.2% (373/688) for RT-PCR positive cases and 71.4% (679/950) for negative cases. Agreement was highest for RT-PCR negative cases labeled as Normal (50.4%, n = 479). Reader performance did not improve with clinical history or time between CXR and RT-PCR result.

Conclusion: At the time of presentation to the emergency department, emergency radiologist performance is non-specific for diagnosing COVID-19.

Keywords: COVID-19; CXR; Chest X-ray; Coronavirus.

MeSH terms

  • Adult
  • Aged
  • COVID-19 Testing
  • COVID-19*
  • Emergency Service, Hospital
  • Female
  • Humans
  • Male
  • Middle Aged
  • Radiography, Thoracic
  • Radiologists
  • Retrospective Studies
  • SARS-CoV-2