Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year period

Jackie Pollack; Wei Yang; Erin M Schnellinger; George J Arnaoutakis; Michael J Kallan; Stephen E Kimmel

doi:10.1016/j.xjon.2023.07.011

Dynamic prediction modeling of postoperative mortality among patients undergoing surgical aortic valve replacement in a statewide cohort over a 12-year period

JTCVS Open. 2023 Jul 24:15:94-112. doi: 10.1016/j.xjon.2023.07.011. eCollection 2023 Sep.

Authors

Jackie Pollack¹, Wei Yang², Erin M Schnellinger³, George J Arnaoutakis⁴, Michael J Kallan⁵, Stephen E Kimmel¹

Affiliations

¹ Department of Epidemiology, College of Public Health and Health Professions and College of Medicine, University of Florida, Gainesville, Fla.
² Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pa.
³ Research Department, United Network For Organ Sharing, Richmond, Va.
⁴ Division of Cardiovascular and Thoracic Surgery, University of Texas at Austin Dell Medical School, Austin, Tex.
⁵ Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pa.

Abstract

Objective: Clinical prediction models for surgical aortic valve replacement mortality, are valuable decision tools but are often limited in their ability to account for changes in medical practice, patient selection, and the risk of outcomes over time. Recent research has identified methods to update models as new data accrue, but their effect on model performance has not been rigorously tested.

Methods: The study population included 44,546 adults who underwent an isolated surgical aortic valve replacement from January 1, 1999, to December 31, 2018, statewide in Pennsylvania. After chronologically splitting the data into training and validation sets, we compared calibration, discrimination, and accuracy measures amongst a nonupdating model to 2 methods of model updating: calibration regression and the novel dynamic logistic state space model.

Results: The risk of mortality decreased significantly during the validation period (P < .01) and the nonupdating model demonstrated poor calibration and reduced accuracy over time. Both updating models maintained better calibration (Hosmer-Lemeshow χ² statistic) than the nonupdating model: nonupdating (156.5), calibration regression (4.9), and dynamic logistic state space model (8.0). Overall accuracy (Brier score) was consistently better across both updating models: dynamic logistic state space model (0.0252), calibration regression (0.0253), and nonupdating (0.0256). Discrimination improved with the dynamic logistic state space model (area under the curve, 0.696) compared with the nonupdating model (area under the curve, 0.685) and calibration regression method (area under the curve, 0.687).

Conclusions: Dynamic model updating can improve model accuracy, discrimination, and calibration. The decision as to which method to use may depend on which measure is most important in each clinical context. Because competing therapies have emerged for valve replacement models, updating may guide clinical decision making.

Keywords: clinical prediction model; dynamic logistic state space model; model recalibration; model updating; surgical aortic valve replacement.