External validation and recalibration of the Brock model to predict probability of cancer in pulmonary nodules using NLST data

Thorax. 2019 Jun;74(6):551-563. doi: 10.1136/thoraxjnl-2018-212413. Epub 2019 Mar 21.

Abstract

Introduction: We performed an external validation of the Brock model using the National Lung Screening Trial (NLST) data set, following strict guidelines set forth by the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement. We report how external validation results can be interpreted and highlight the role of recalibration and model updating.

Materials and methods: We assessed model discrimination and calibration using the NLST data set. Adhering to the inclusion/exclusion criteria reported by McWilliams et al, we identified 7879 non-calcified nodules discovered at the baseline low-dose CT screen with 2 years of follow-up. We characterised differences between Pan-Canadian Early Detection of Lung Cancer Study and NLST cohorts. We calculated the slope on the prognostic index and the intercept coefficient by fitting the original Brock model to NLST. We also assessed the impact of model recalibration and the addition of new covariates such as body mass index, smoking status, pack-years and asbestos.

Results: While the area under the curve (AUC) of the model was good, 0.905 (95% CI 0.882 to 0.928), a histogram plot showed that the model poorly differentiated between benign and malignant cases. The calibration plot showed that the model overestimated the probability of cancer. In recalibrating the model, the coefficients for emphysema, spiculation and nodule count were updated. The updated model had an improved calibration and achieved an optimism-corrected AUC of 0.912 (95% CI 0.891 to 0.932). Only pack-year history was found to be significant (p<0.01) among the new covariates evaluated.

Conclusion: While the Brock model achieved a high AUC when validated on the NLST data set, the model benefited from updating and recalibration. Nevertheless, covariates used in the model appear to be insufficient to adequately discriminate malignant cases.

Keywords: Brock model; external validation; lung cancer; prediction; recalibration.

Publication types

  • Research Support, N.I.H., Extramural
  • Research Support, Non-U.S. Gov't
  • Validation Study

MeSH terms

  • Aged
  • Calibration
  • Datasets as Topic
  • Early Detection of Cancer
  • Female
  • Guideline Adherence
  • Humans
  • Lung Neoplasms / diagnostic imaging*
  • Lung Neoplasms / pathology
  • Male
  • Mass Screening*
  • Middle Aged
  • Models, Statistical
  • Multiple Pulmonary Nodules / diagnostic imaging*
  • Multiple Pulmonary Nodules / pathology
  • Predictive Value of Tests
  • Probability
  • Prognosis
  • Randomized Controlled Trials as Topic
  • Risk Assessment / methods
  • Solitary Pulmonary Nodule / diagnostic imaging*
  • Solitary Pulmonary Nodule / pathology
  • Tomography, X-Ray Computed*