Skip to main page content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
, 36 (21), 3283-3301

Measuring the Statistical Validity of Summary Meta-Analysis and Meta-Regression Results for Use in Clinical Practice


Measuring the Statistical Validity of Summary Meta-Analysis and Meta-Regression Results for Use in Clinical Practice

Brian H Willis et al. Stat Med.


An important question for clinicians appraising a meta-analysis is: are the findings likely to be valid in their own practice-does the reported effect accurately represent the effect that would occur in their own clinical population? To this end we advance the concept of statistical validity-where the parameter being estimated equals the corresponding parameter for a new independent study. Using a simple ('leave-one-out') cross-validation technique, we demonstrate how we may test meta-analysis estimates for statistical validity using a new validation statistic, Vn, and derive its distribution. We compare this with the usual approach of investigating heterogeneity in meta-analyses and demonstrate the link between statistical validity and homogeneity. Using a simulation study, the properties of Vn and the Q statistic are compared for univariate random effects meta-analysis and a tailored meta-regression model, where information from the setting (included as model covariates) is used to calibrate the summary estimate to the setting of application. Their properties are found to be similar when there are 50 studies or more, but for fewer studies Vn has greater power but a higher type 1 error rate than Q. The power and type 1 error rate of Vn are also shown to depend on the within-study variance, between-study variance, study sample size, and the number of studies in the meta-analysis. Finally, we apply Vn to two published meta-analyses and conclude that it usefully augments standard methods when deciding upon the likely validity of summary meta-analysis estimates in clinical practice. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Keywords: data interpretation; decision making; meta-analysis; models; statistical; validity.


Figure 1
Figure 1
Power of VnMeta‐analysis in left panel and meta‐regression in right. In both panels, we have τ varying, σ = 1, and n = 100.
Figure 2
Figure 2
Comparison of power of Vn and Q for meta‐analysis and meta‐regression (with 1 covariate).

Similar articles

See all similar articles

Cited by 8 PubMed Central articles

See all "Cited by" articles


    1. Riley RD, Ahmed I, Debray TPA, Willis BH, Noordzij P, Higgins JPT, Deeks JJ. Summarising and validating the accuracy of a diagnostic or prognostic test across multiple studies: a new meta‐analysis framework. Statistics in Medicine 2015; 34:2081–2103. - PMC - PubMed
    1. Debray TP, Moons KG, Ahmed I, Koffijberg H, Riley RD. A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta‐analysis. Statistics in Medicine 2013; 32:3158–3180. - PubMed
    1. Snell KIE, Hua H, Ensor J, Debray TP, Look MP, Moons KG, Riley RD. Multivariate meta‐analysis of individual participant data helped externally validate the performance and implementation of a prediction model. Journal of Clinical Epidemiology 2016; 69:40–50. - PMC - PubMed
    1. Royston P, Parmar MKB, Sylvester R. Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer. Statistics in Medicine 2004; 23:907–926. - PubMed
    1. Willis BH, Hyde CJ. Estimating a test's accuracy using tailored meta‐analysis—how setting‐specific data may aid study selection. Journal of Clinical Epidemiology 2014; 67:538–546. - PubMed

LinkOut - more resources