Derivation and reliability of an instrument to estimate medical benefit of emergency treatment

Abstract

Objectives: For many patients, it is difficult to define the benefit derived from a visit to the emergency department (ED). No criterion standard exists that defines benefit from emergency treatment compared to routine outpatient care, and our limited ability to estimate benefit from emergency treatment has significant implications for emergency care-related health services research. The objectives of this study were to develop a decision algorithm to be used in estimating benefit of emergency treatment (EBET) and to assess its reliability when applied to patients making unscheduled ED visits.

Methods: The EBET instrument defines benefit as a 3-level outcome, namely, significant, possible, or unlikely, and its content validity was assessed through expert review. The instrument was independently applied by multiple investigators to 3 different ED patient cohorts. A consensus-based process was used to determine the final EBET for each patient visit. Weighted kappas and their 95% confidence intervals were calculated to assess the reliability of the EBET Instrument applied individually, and the Spearman-Brown formula was used to assess the overall reliability of the EBET instrument when applied using multiple raters and a standardized consensus process.

Results: A total of 875 visits (300 from a general ED population, 300 from a homeless ED population, and 275 from an HIV-infected ED population) were scored using the EBET instrument. The consensus process included independently scoring groups ranging from approximately 50 to 100 patient visits, determining the level of agreement, discussing the discordant results among the investigators, and assigning a final EBET category to each visit. This process was repeated sequentially until all visits within each cohort were scored. The overall weighted kappas ranged from 0.66 to 0.76, and the Spearman-Brown correlation ranged from 0.83 to 0.87.

Conclusions: The EBET instrument demonstrated good to excellent reliability when applied independently by raters to both unselected and selected ED patients. Its reliability, however, was excellent to outstanding when multiple raters applied it using a consensus process. The EBET instrument may serve as a useful tool for defining benefit from emergency treatment.