A combined pre-clinical meta-analysis and randomized confirmatory trial approach to improve data validity for therapeutic target validation

Abstract

Biomedical research suffers from a dramatically poor translational success. For example, in ischemic stroke, a condition with a high medical need, over a thousand experimental drug targets were unsuccessful. Here, we adopt methods from clinical research for a late-stage pre-clinical meta-analysis (MA) and randomized confirmatory trial (pRCT) approach. A profound body of literature suggests NOX2 to be a major therapeutic target in stroke. Systematic review and MA of all available NOX2(-/y) studies revealed a positive publication bias and lack of statistical power to detect a relevant reduction in infarct size. A fully powered multi-center pRCT rejects NOX2 as a target to improve neurofunctional outcomes or achieve a translationally relevant infarct size reduction. Thus stringent statistical thresholds, reporting negative data and a MA-pRCT approach can ensure biomedical data validity and overcome risks of bias.

Figure 1. Risk of bias and reporting quality of NOX2 studies, averaged per item.

Items, i.e. risks of bias are listed on the left. Open bars indicate a low risk of bias; closed, a high risk; hatched bars, an unclear risk. Items 1 and 3 scored reporting.

Figure 2. Meta-analysis of the overall effect of NOX2 on infarct size and neurological score in experimental stroke.

Studies included are shown on the left and analyzed in two forest plots, either without (left section) or with (right section) the data of the here presented randomized, confirmatory, blinded study. Subgroups within one study are depicted separately with the following coding: a, female gender; b, male gender; c, short ischemic time; d, medium ischemic time; e, long ischemic time. The upper half a, contains data for the effect of NOX2 on infarct size (IS); the lower half b, on neurological score (NS). Displayed are the standardized mean difference (SMD), 95% confidence intervals and relative weight of the individual studies. The diamond indicates the global SMD and its 95% confidence interval.

Figure 3. Funnel plot asymmetry suggesting the presence of publication bias and an overestimation of the overall effect size of NOX2 in stroke.

The Y-axis represents precision; the X-axis effect size of individual studies. The funnel plot is based on the fact that precision in estimating the underlying treatment effect will increase as the sample size of component studies increases. Using the trim and fill analyses the intervention effect is adjusted for possible missing studies (filled red symbols) amongst published data (open symbols). The asymmetry suggests that studies showing larger infarcts in NOX2 KO mice in experimental stroke are missing. These would otherwise shift the mean (open diamond) towards a smaller or no overall effect of NOX2 (closed diamond).

Figure 4. Infarct size and neurological function in NOX2 and WT mice.

Mice of both genders and between 6–20 weeks old were subjected to tMCAO. (A) TTC stainings of three sequential coronal brain sections on day 1 after tMCAO that were representative for each subgroup: young (6–10 week old), adult (18–20 week old) male, and adult female NOX2 KO and WT mice. The TTC staining colors viable tissue red, while infarcted tissue stays white. (B) Bar graphs of mean infarct volumes ± SEM from WT (open bar) and NOX2 KO mice (closed bars). (C) Scatter plot and median of neurodeficit Bederson scores on day 1 after tMCAO, ranging from 0 (normal) to 5 (severe), from WT (open symbols) and NOX2 KO mice (closed symbols). (D) Scatter plot and median of grip test scores, ranging from 0 (severe deficit) to 5 (normal) from WT (open symbols) and NOX2 KO mice (closed symbols). The asterisk, *, indicates statistical significance (P < 0.05, t-test) for the infarct size in the subgroup adult male NOX2 versus WT; however, this group was insufficiently powered to allow the detection of a difference. Combined analysis did not show a significant difference for any of the parameters.

Figure 5. Updated funnel plot with the data from the current trial.

Funnel plot asymmetry still suggests the presence of publication bias and an overestimation of the overall effect size of NOX2 in stroke. Using the trim and fill analyses, the intervention effect is adjusted for possible missing studies (filled red symbols) amongst published and present data (open symbols). Studies showing larger infarcts in NOX2 KO mice and thus a smaller or no effect of NOX2 in experimental stroke, are still missing. For further explanations see Fig. 3.