Unit of analysis issues in laboratory-based research

Abstract

Many studies in the biomedical research literature report analyses that fail to recognise important data dependencies from multilevel or complex experimental designs. Statistical inferences resulting from such analyses are unlikely to be valid and are often potentially highly misleading. Failure to recognise this as a problem is often referred to in the statistical literature as a unit of analysis (UoA) issue. Here, by analysing two example datasets in a simulation study, we demonstrate the impact of UoA issues on study efficiency and estimation bias, and highlight where errors in analysis can occur. We also provide code (written in R) as a resource to help researchers undertake their own statistical analyses.

Keywords: Science Forum; epidemiology; experimental design; global health; mixed-effects models; statistics.

Figure 1.. A strip plot showing observed lymph node size data by subject (1-12) and sample, after none and a short course of radiotherapy (Short RT).

Figure 2.. Boxplots of residuals (observed values - fitted values) for each subject; symbols (∙) are medians, boxes are interquartile ranges (IQR), whiskers extend to 1.5×IQR and symbols (∘) outside these are suspected outliers (a).

Quantile-quantile (Q–Q) plot of the model residuals ($\circ$) on the horizontal axis against theoretical residuals from a Normal distribution on the vertical axis (b).

Appendix 1—figure 1.. Naive use of a conventional t-test on correlated (grouped by subject) data, ρ = 0 (black circle ), ρ = 0.2 (red circle) ρ = 0.5 (blue circle) and ρ = 0.8 (green circle), inflates the type I error rate (set at 5%).

(a). The type I error rate can be controlled to the required level by randomly selecting a single measurement for each subject, $\rho$ = 0 (black circle), $\rho$ = 0.2 (red circle), $\rho$ = 0.5 (blue circle) and $\rho$ = 0.8 (green circle), or made conservative ($\le$5%) by taking the mean of the measurements for each subject, $\rho$ = 0 (black open circle), $\rho$ = 0.2 (red open circle), $\rho$ = 0.5 (blue open circle) and $\rho$ = 0.8 (green open circle) (b). The relative efficiency of treatment effect estimates declines as the number of clusters become smaller and is always higher for the mean than the randomly selected single measurement strategy (c). The scenarios (i) – (vi) are as described in the text.

Appendix 2—figure 1.. Design options for a putative laboratory study testing n samples of experimental material.