High certainty evidence is stable and trustworthy, whereas evidence of moderate or lower certainty may be equally prone to being unstable

Benjamin Djulbegovic; Despina Koletsi; Iztok Hozo; Amy Price; Ana Luiza Cabrera Martimbianco; Rachel Riera; Paulo Nadanovsky; Ana Paula Pires Dos Santos; Nikolaos Pandis; Rafael Leite Pacheco; Luis Eduardo Fontes; Jadbinder Seehra; Muneeb Ahmed; Liang Yao; David Nunan; Lars G Hemkens

doi:10.1016/j.jclinepi.2024.111392

High certainty evidence is stable and trustworthy, whereas evidence of moderate or lower certainty may be equally prone to being unstable

J Clin Epidemiol. 2024 May 11:111392. doi: 10.1016/j.jclinepi.2024.111392. Online ahead of print.

Authors

Benjamin Djulbegovic¹, Despina Koletsi², Iztok Hozo³, Amy Price⁴, Ana Luiza Cabrera Martimbianco⁵, Rachel Riera⁶, Paulo Nadanovsky⁷, Ana Paula Pires Dos Santos⁸, Nikolaos Pandis⁹, Rafael Leite Pacheco⁶, Luis Eduardo Fontes¹⁰, Jadbinder Seehra¹¹, Muneeb Ahmed¹², Liang Yao¹³, David Nunan¹⁴, Lars G Hemkens¹⁵

Affiliations

¹ Division of Hematology/Oncology, Department of Medicine, Medical University of South Carolina, Charleston, SC, USA.
² Clinic of Orthodontics and Pediatric Dentistry, Center of Dental Medicine, University of Zurich, Zurich, Switzerland; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA. Electronic address: despoina.koletsi@zzm.uzh.ch.
³ Department of Mathematics, Indiana University Northwest, Gary, Indiana, USA.
⁴ Anesthesia Informatics and Media Lab, Stanford University, Stanford, California, USA.
⁵ Centre of Health Technology Assessment, Hospital Sírio-Libanês, São Paulo, Brazil; Postgraduate Program of Health and Environment, Universidade Metropolitana de Santos, Brazil.
⁶ Centre of Health Technology Assessment, Hospital Sírio-Libanês, São Paulo, Brazil; Universidade Federal de São Paulo, Escola Paulista de Medicina, Brazil (Unifesp), São Paulo, Brazil.
⁷ Department of Epidemiology and Quantitative Methods in Health, National School of Public Health, Fundação Oswaldo Cruz, Rio de Janeiro, Brazil; Department of Epidemiology, Institute of Social Medicine, Universidade do Estado do Rio de Janeiro, Brazil.
⁸ Department of Community and Preventive Dentistry, Faculty of Dentistry, Universidade do Estado do Rio de Janeiro, Brazil.
⁹ Department of Orthodontics and Dentofacial Orthopedics, Dental School/Medical Faculty, University of Bern, Bern, Switzerland.
¹⁰ Department of Intensive Care, and Emergency Medicine at Faculdade de Medicina de Petrópolis, in Petrópolis, Rio de Janeiro, Brazil.
¹¹ Centre for Craniofacial Development & Regeneration, Faculty of Dentistry, Oral & Craniofacial Sciences, King's College London, Floor 25, Guy's Hospital, London, SE1 9RT, United Kingdom.
¹² Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada.
¹³ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada.
¹⁴ Kellogg College, University of Oxford, Oxford, UK; Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, Oxford University, Oxford, UK.
¹⁵ Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA; Department of Clinical Research, University of Basel, University Hospital Basel, Switzerland; Pragmatic Evidence Lab, Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB), University Hospital Basel and University of Basel, Basel, Switzerland; Meta-Research Innovation Center Berlin (METRIC-B), Berlin Institute of Health, Berlin, Germany.

PMID: 38740313
DOI: 10.1016/j.jclinepi.2024.111392

Abstract

Objective: To assess to what extent the overall quality of evidence indicates changes to observed intervention effect estimates when new data become available.

Study design and setting: We conducted a meta- epidemiological study. We obtained evidence from meta-analyses of randomized trials of Cochrane reviews addressing the same healthcare question that was updated with inclusion of additional data between January 2016 and May 2021.

Methods: We extracted the reported effect estimates with 95% confidence intervals from meta-analyses and corresponding GRADE (Grading of Recommendations Assessment, Development, and Evaluation) assessments of any intervention comparison for the primary outcome in the first and the last updated review version. We considered the reported overall quality (certainty) of evidence (CoE) and specific evidence limitations (no, serious or very serious for risk of bias, imprecision, inconsistency, and/or indirectness). We assessed the change in pooled effect estimates between the original and updated evidence using the ratio of odds ratio (ROR), absolute ROR (aROR), ratio of standard errors (RoSE), direction of effects, and level of statistical significance.

Results: High CoE without limitations characterized 19.3% (n=29) out of 150 included original Cochrane reviews. The update with additional data did not systematically change the effect estimates (mean ROR 1.00; 95%CI 0.99-1.02), which deviated 1.06-fold from the older estimates (median aROR; IQR: 1.01-1.15), gained precision (median RoSE 0.87; IQR 0.76-1.00), and maintained the same direction with the same level of statistical significance in 93% (27 of 29) of cases. Lower CoE with limitations characterized 121 original reviews and graded as moderate CoE in 30.0% (45 of 150), low CoE in 32.0% (48 of 150), and very low CoE in 18.7% (28 of 150) reviews. Their update had larger absolute deviations (median aROR 1.12 to 1.33) and larger gains in precision (median RoSE 0.78 to 0.86) without clear and consistent differences between these categories of CoE. Changes in effect direction or statistical significance were also more common in the lower quality evidence, again with a similar extent across categories (without change in 75.6%, 64.6%, and 75.0% for moderate, low, very low CoE). As limitations increased, effect estimates deviated more (aROR 1.05 with zero, 1.11 with one, 1.25 with two, 1.24 with three limitations) and changes in direction or significance became more frequent (93.2% stable with no limitations, 74.5% with one, 68.2% with two, and 61.5% with three limitations).

Conclusions: High-quality evidence without methodological deficiencies is trustworthy and stable, providing reliable intervention effect estimates when updated with new data. Evidence of moderate and lower quality may be equally prone to being unstable and cannot indicate if available effect estimates are true, exaggerated, or underestimated.

Keywords: GRADE; bias; critical appraisal; evidence-based medicine; meta-epidemiology; systematic review.