Interobserver Reliability of the Berlin ARDS Definition and Strategies to Improve the Reliability of ARDS Diagnosis

Abstract

Background: Failure to reliably diagnose ARDS may be a major driver of negative clinical trials and underrecognition and treatment in clinical practice. We sought to examine the interobserver reliability of the Berlin ARDS definition and examine strategies for improving the reliability of ARDS diagnosis.

Methods: Two hundred five patients with hypoxic respiratory failure from four ICUs were reviewed independently by three clinicians, who evaluated whether patients had ARDS, the diagnostic confidence of the reviewers, whether patients met individual ARDS criteria, and the time when criteria were met.

Results: Interobserver reliability of an ARDS diagnosis was "moderate" (kappa = 0.50; 95% CI, 0.40-0.59). Sixty-seven percent of diagnostic disagreements between clinicians reviewing the same patient was explained by differences in how chest imaging studies were interpreted, with other ARDS criteria contributing less (identification of ARDS risk factor, 15%; cardiac edema/volume overload exclusion, 7%). Combining the independent reviews of three clinicians can increase reliability to "substantial" (kappa = 0.75; 95% CI, 0.68-0.80). When a clinician diagnosed ARDS with "high confidence," all other clinicians agreed with the diagnosis in 72% of reviews. There was close agreement between clinicians about the time when a patient met all ARDS criteria if ARDS developed within the first 48 hours of hospitalization (median difference, 5 hours).

Conclusions: The reliability of the Berlin ARDS definition is moderate, driven primarily by differences in chest imaging interpretation. Combining independent reviews by multiple clinicians or improving methods to identify bilateral infiltrates on chest imaging are important strategies for improving the reliability of ARDS diagnosis.

Keywords: ARDS; acute lung injury; clinical trials; diagnosis.

Figure 1

Interobserver reliability between clinicians applying the Berlin ARDS definition to a cohort of 205 patients with acute hypoxic respiratory failure. A, Interobserver reliability for ARDS diagnosis within 24 hours after the onset of invasive mechanical ventilation and at any point during the hospitalization. B, Additional measures of agreement. All patients were reviewed in triplicate, and reliability was calculated using Cohen’s kappa for multiple nonunique reviewers. Prevalence is the proportion of reviews in which ARDS was present. Raw agreement is the overall rate of agreement between clinicians. Positive agreement is the rate of agreement among patients believed to have acquired ARDS. Negative agreement is the rate of agreement among patients who were thought not to have acquired ARDS.

Figure 2

Interobserver agreement between two individual clinicians applying the Berlin ARDS definition and the interobserver agreement between two groups of three clinicians. In this approach, individuals perform ARDS reviews independently, and the group assessment is the combined average of three clinicians’ individual assessments. Interobserver agreement is calculated using the intraclass correlation coefficient.

Figure 3

Relationship between an individual clinician’s confidence in the diagnosis of ARDS and the assessment of other clinicians.

Figure 4

Potential for improvement in the reliability of ARDS diagnosis after improvements in individual ARDS criteria. Improvement in the reliability of individual ARDS criteria on the effect on ARDS diagnosis was simulated with assumption details described in e-Appendix 1. Absolute improvement in the reliability of ARDS diagnosis is calculated as the difference in the reliability of ARDS diagnosis before and after the reliability of the individual ARDS criteria was improved.