A decade of test-retest reliability of functional connectivity: A systematic review and meta-analysis

Abstract

Background: Once considered mere noise, fMRI-based functional connectivity has become a major neuroscience tool in part due to early studies demonstrating its reliability. These fundamental studies revealed only the tip of the iceberg; over the past decade, many test-retest reliability studies have continued to add nuance to our understanding of this complex topic. A summary of these diverse and at times contradictory perspectives is needed.

Objectives: We aimed to summarize the existing knowledge regarding test-retest reliability of functional connectivity at the most basic unit of analysis: the individual edge level. This entailed (1) a meta-analytic estimate of reliability and (2) a review of factors influencing reliability.

Methods: A search of Scopus was conducted to identify studies that estimated edge-level test-retest reliability. To facilitate comparisons across studies, eligibility was restricted to studies measuring reliability via the intraclass correlation coefficient (ICC). The meta-analysis included a random effects pooled estimate of mean edge-level ICC, with studies nested within datasets. The review included a narrative summary of factors influencing edge-level ICC.

Results: From an initial pool of 212 studies, 44 studies were identified for the qualitative review and 25 studies for quantitative meta-analysis. On average, individual edges exhibited a "poor" ICC of 0.29 (95% CI = 0.23 to 0.36). The most reliable measurements tended to involve: (1) stronger, within-network, cortical edges, (2) eyes open, awake, and active recordings, (3) more within-subject data, (4) shorter test-retest intervals, (5) no artifact correction (likely due in part to reliable artifact), and (6) full correlation-based connectivity with shrinkage.

Conclusion: This study represents the first meta-analysis and systematic review investigating test-retest reliability of edge-level functional connectivity. Key findings suggest there is room for improvement, but care should be taken to avoid promoting reliability at the expense of validity. By pooling existing knowledge regarding this key facet of accuracy, this study supports broader efforts to improve inferences in the field.

Keywords: Edge-level; Intraclass correlation coefficient; Systematic review; Validity; fMRI.

Figure 1.. PRISMA flow diagram summarizing publication selection for review and meta-analysis.

“Identification” refers to initial search; “screening” refers to removal of records based on title and abstract; “eligibility” refers to removal of records based on full-text; “included” refers to inclusion of studies for qualitative and quantitative analysis. From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group(2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Plos Med 6(7): e000097. Doi:10.137/journal.p.med1000097

Figure 2.. Summary of reliability across studies.

Forest plot depicting mean ICCs and 95% confidence intervals for the 25 studies included in the meta-analysis. The pooled estimate of the ICC was obtained using a random effects (RE) model with studies nested by dataset.

Figure 3.. Similarity of adjacent sessions, separated by approximately one week.

Using the Yale Test-Retest dataset (Noble et al., 2017b), similarity was measured using the Pearson’s correlation between connectomes assessed about a week apart. A, Correlations between session pairs, averaged across subjects. B, Correlations between session pairs for each subject plotted as a function of inter-session interval. Lines indicate each subject, with colors indicating session pairs. The group level correlation is displayed in the bottom left.

Figure 4.. Correlation between variance components and ICC.

Variance components were calculated using the Yale Test-Retest dataset as described in Noble et al. (2017b). Each panel corresponds with a variance component and demonstrates the correlation between that variance component and ICC. Blue dots indicate “strong” edges (p<0.05), and red dots indicate “weak” edges (p>0.05). ICC is most correlated with variance between subjects, and second most correlated with overall variance (r=0.79; not shown).