Increased risks for random errors are common in outcomes graded as high certainty of evidence

Abstract

Objectives: The aim of article was to assess the risk for random errors in outcomes graded as high certainty of evidence (CoE).

Study design and setting: We randomly selected 100 Cochrane reviews with dichotomous outcomes rated as high CoE using Grading of Recommendations Assessment, Development, and Evaluation. To detect increased risks for random errors, two investigators independently conducted trial sequential analysis using conventional thresholds for type I (α = 0.05) and type II (β = 0.10) errors. We dually regraded all outcomes with increased risks for random errors and conducted multivariate logistic regression analyses to determine predictors of increased risks for random errors.

Results: Overall, 38% (95% confidence interval: 28-47%) of high CoE outcomes had increased risks for random errors. Outcomes assessing harms were more frequently affected than outcomes assessing benefits (47% vs. 12%). Regrading of outcomes with increased random errors showed that 74% should have been downgraded based on current guidance. Regression analyses rendered small absolute risk differences (P = 0.009) and low number of events (P = 0.001) as significant predictors of increased risks for random errors.

Conclusion: Decisionmakers need to be aware that outcomes rated as high CoE often have increased risks for false-positive or false-negative findings.

Keywords: Decision-making; GRADE; Random errors; Systematic reviews; Trial sequential analysis.