Performance of the streamlined quality outcomes database web-based calculator: internal and external validation

Abstract

Background context: With an increasing number of web-based calculators designed to provide the probabilities of an individual achieving improvement after lumbar spine surgery, there is a need to determine the accuracy of these models.

Purpose: To perform an internal and external validation study of the reduced Quality Outcomes Database web-based Calculator (QOD-Calc).

Study design: Observational longitudinal cohort.

Patient sample: Patients enrolled study-wide in Quality Outcomes Database (QOD) and patients enrolled in DaneSpine at a single institution who had elective lumbar spine surgery with baseline data to complete QOD-Calc and 12-month postoperative data.

Outcome measures: Oswestry Disability Index (ODI), Numeric Rating Scales (NRS) for back and leg pain, EuroQOL-5D (EQ-5D).

Methods: Baseline data elements were entered into QOD-Calc to determine the probability for each patient having Any Improvement and 30% Improvement in NRS leg pain, back pain, EQ-5D and ODI. These probabilities were compared with the actual 12-month postop data for each of the QOD and DaneSpine cases. Receiver-operating characteristics analyses were performed and calibration plots created to assess model performance.

Results: 24,755 QOD cases and 8,105 DaneSpine lumbar cases were included in the analysis. QOD-Calc had acceptable to outstanding ability (AUC: 0.694-0.874) to predict Any Improvement in the QOD cohort and moderate to acceptable ability (AUC: 0.658-0.747) to predict 30% Improvement. QOD-Calc had acceptable to exceptional ability (AUC: 0.669-0.734) to predict Any improvement and moderate to exceptional ability (AUC: 0.619-0.862) to predict 30% Improvement in the DaneSpine cohort. AUCs for the DaneSpine cohort was consistently lower that the AUCs for the QOD validation cohort.

Conclusion: QOD-Calc performs well in predicting outcomes in a patient population that is similar to the patients that was used to develop it. Although still acceptable, model performance was slightly worse in a distinct population, despite the fact that the sample was more homogenous. Model performance may also be attributed to the low discrimination threshold, with close to 90% of cases reporting Any Improvement in outcome. Prediction models may need to be developed that are highly specific to the characteristics of the population.

Keywords: Back pain; External validation; Internal validation; Leg pain; Lumbar spine surgery; Oswestry Disability Index; Predictive model; Web-based model.