Generalizability of findings from a clinical sample to a community-based sample: A comparison of ADNI and ARIC

Abstract

Introduction: Clinic-based study samples, including the Alzheimer's Disease Neuroimaging Initiative (ADNI), offer rich data, but findings may not generalize to community-based settings. We compared associations in ADNI to those in the Atherosclerosis Risk in Communities (ARIC) study to assess generalizability across the two settings.

Methods: We estimated cohort-specific associations among risk factors, cognitive test scores, and neuroimaging outcomes to identify and quantify the extent of significant and substantively meaningful differences in associations between cohorts. We explored whether using more homogenous samples improved comparability in effect estimates.

Results: The proportion of associations that differed significantly between cohorts ranged from 27% to 34% across sample subsets. Many differences were substantively meaningful (e.g., odds ratios [OR] for apolipoprotein E ε4 on amyloid positivity in ARIC: OR = 2.8, in ADNI: OR = 8.6).

Discussion: A higher proportion of associations differed significantly and substantively than would be expected by chance. Findings in clinical samples should be confirmed in more representative samples.

Keywords: Alzheimer's Disease Neuroimaging Initiative; Alzheimer's disease; Atherosclerosis Risk in Communities; dementia; generalizability.

Figure 1.

Heatmap of P-values for statistical significance in differences across ADNI and ARIC estimated associations between risk factors and cognitive test/imaging outcomes, and between imaging outcomes and cognitive test outcomes; limited to associations with p<0.05 statistical significance independently in at least one study NA: Association not applicable to sample I: Association not significant to p<0.05 in either cohort

Figure 2.

Heatmap of percentage differences in point estimates of associations^a between risk factors and cognitive test/imaging outcomes, and between imaging outcomes and cognitive test outcomes; limited to associations with p<0.05 statistical significance independently in at least one study ^a Percentage difference relative to ARIC estimates, computed as ∣ARIC estimate – ADNI estimate∣/∣ARIC estimate∣ NA: Association not applicable to sample I: Association not significant to p<0.05 in either cohort

Figure 3.

Comparison of estimated associations in ADNI vs. in ARIC using all available observations ^a that are significant to p<0.05 in at least one cohort; continuous outcomes ^a for associations involving SUVR, this excludes observations of participants with dementia

Figure 4.

Comparison of estimated associations in ADNI vs. in ARIC using all available observations ^a that are significant to p<0.05 in at least one cohort; binary outcomes ^a for associations involving SUVR, this excludes observations of participants with dementia ^b MMSE (score < 25) ^c boston naming (score < 25) ^d SUVR (cohort cut-off)