GRADE concept paper 8: judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies

Farid Foroutan; Martin Mayer; Gordon Guyatt; Richard D Riley; Reem Mustafa; Nina Kreuzberger; Nicole Skoetz; Andrea Darzi; Ana Carolina Alba; Fabrice Mowbray; Daniel G Rayner; Holger Schunemann; Alfonso Iorio

doi:10.1016/j.jclinepi.2024.111344

GRADE concept paper 8: judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies

J Clin Epidemiol. 2024 Apr 3:170:111344. doi: 10.1016/j.jclinepi.2024.111344. Online ahead of print.

Authors

Affiliations

¹ Ted Rogers Centre for Heart Research, University Health Network, Toronto, Ontario, Canada; Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada. Electronic address: Farid.foroutan@uhn.ca.
² DynaMed Decisions, EBSCO Clinical Decisions, EBSCO, Ipswich, MA, USA; Open Door Clinic, Cone Health, Greensboro, NC, USA.
³ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Medicine, McMaster University, Hamilton, Ontario, Canada.
⁴ National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, England, UK; Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
⁵ Division of Nephrology and Hypertension, Department of Medicine, University of Kansas School of Medicine, Kansas City, MO, USA.
⁶ Evidence-Based Medicine, Department of Internal Medicine, Faculty of Medicine and University Hospital Cologne, University of Cologne, Cologne, Germany.
⁷ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada.
⁸ Ted Rogers Centre for Heart Research, University Health Network, Toronto, Ontario, Canada; Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada.
⁹ College of Nursing, Michigan State University, Kansas City, MI, USA.
¹⁰ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada.

PMID: 38579978
DOI: 10.1016/j.jclinepi.2024.111344

Abstract

Background: Prognostic models incorporate multiple prognostic factors to estimate the likelihood of future events for individual patients based on their prognostic factor values. Evaluating these models crucially involves conducting studies to assess their predictive performance, like discrimination. Systematic reviews and meta-analyses of these validation studies play an essential role in selecting models for clinical practice.

Methods: In this paper, we outline 3 thresholds to determine the target for certainty rating in the discrimination of prognostic models, as observed across a body of validation studies.

Results and conclusion: We propose 3 thresholds when rating the certainty of evidence about a prognostic model's discrimination. The first threshold amounts to rating certainty in the model's ability to classify better than random chance. The other 2 approaches involve setting thresholds informed by other mechanisms for classification: clinician intuition or an alternative prognostic model developed for the same disease area and outcome. The choice of threshold will vary based on the context. Instead of relying on arbitrary discrimination cut-offs, our approach positions the observed discrimination within an informed spectrum, potentially aiding decisions about a prognostic model's practical utility.

Keywords: Certainty in evidence; Discrimination; GRADE; Meta-analysis; Prognosis; Prognostic models; Systematic review.