Mitigating site effects in covariance for machine learning in neuroimaging data

Abstract

To acquire larger samples for answering complex questions in neuroscience, researchers have increasingly turned to multi-site neuroimaging studies. However, these studies are hindered by differences in images acquired across multiple sites. These effects have been shown to bias comparison between sites, mask biologically meaningful associations, and even introduce spurious associations. To address this, the field has focused on harmonizing data by removing site-related effects in the mean and variance of measurements. Contemporaneously with the increase in popularity of multi-center imaging, the use of machine learning (ML) in neuroimaging has also become commonplace. These approaches have been shown to provide improved sensitivity, specificity, and power due to their modeling the joint relationship across measurements in the brain. In this work, we demonstrate that methods for removing site effects in mean and variance may not be sufficient for ML. This stems from the fact that such methods fail to address how correlations between measurements can vary across sites. Data from the Alzheimer's Disease Neuroimaging Initiative is used to show that considerable differences in covariance exist across sites and that popular harmonization techniques do not address this issue. We then propose a novel harmonization method called Correcting Covariance Batch Effects (CovBat) that removes site effects in mean, variance, and covariance. We apply CovBat and show that within-site correlation matrices are successfully harmonized. Furthermore, we find that ML methods are unable to distinguish scanner manufacturer after our proposed harmonization is applied, and that the CovBat-harmonized data retain accurate prediction of disease group.

Keywords: ComBat; cortical thickness; covariance; harmonization; multi-site analysis; site effect.

FIGURE 1

Average across sites of the Frobenius distance between sample site‐specific covariance matrices and the true covariance matrix for the Simple Covariance Effects simulation. The displayed values are averaged across the mean Frobenius distance for each site, which are taken across 1,000 simulations each. Results are plotted for a sample size per site of 25, 50, 100, 250, 500, 1,000, 5,000, and 10,000

FIGURE 2

Results from ML simulations for detection of site and for detection of the simulated diagnosis in the absence of simulated diagnosis effects on covariance. The simulated data consists of 62 cortical thicknesses for 250 subjects per site across three sites. For each of 1,000 simulations, the data is randomly split into 50% training and 50% validation. A random forests algorithm is trained using the training set to predict either Site 1 or the presence of the simulated diagnosis. (a), Boxplot showing Site 1 detection in the ComBat simulation. (b), Boxplot showing simulated diagnosis detection in the ComBat simulation. (c), Boxplot showing Site 1 detection in the diagnosis affects mean simulation. (d), Boxplot showing simulated diagnosis detection in the diagnosis affects mean simulation

FIGURE 3

Results from ML simulations for detection of site and for detection of the simulated diagnosis where the simulated diagnosis also confounds covariance. The simulated data consists of 62 cortical thicknesses for 250 subjects per site across three sites. For each of 1,000 simulations, the data is randomly split into 50% training and 50% validation. A random forests algorithm is trained using the training set to predict either Site 1 or the presence of the simulated diagnosis. (a), Boxplot showing Site 1 detection in the Diagnosis Affects Covariance simulation. (b), Boxplot showing simulated diagnosis detection in the diagnosis affects covariance simulation. (c), Boxplot showing Site 1 detection in the covariance only simulation. (d), Boxplot showing simulated diagnosis detection in the Covariance Only simulation

FIGURE 4

Covariance matrices for cortical thickness values acquired on three sites before and after harmonization. All covariance matrices are estimated after residualizing the data on age, sex, and diagnosis status. Site A uses a Siemens Symphony 1.5T scanner with 23 subjects and the other sites use General Electric Signa Excite 1.5T scanners with 20 subjects each

FIGURE 5

Multivariate pattern analysis experiments for detection of scanner manufacturer, sex, and Alzheimer's disease status using cortical thickness data. The data are randomly split into 50% training and 50% validation then used to train a random forests algorithm to predict a specified trait. AUC values from 100 repetitions of this analysis are reported for unharmonized, ComBat‐adjusted, and CovBat‐adjusted data. (a) Boxplot showing results for detecting if subjects were acquired on a Siemens scanner. Results for detection of Alzheimer's disease status are shown in (b) and results for detection of sex are shown in (c)

FIGURE A1

Multivariate pattern analysis experiments for detection of age using cortical thickness data. The data is randomly split into 50% training and 50% validation then used to train a random forests algorithm to predict age. RMSE values from 100 repetitions of this analysis are reported for unharmonized, ComBat‐adjusted, and CovBat‐adjusted data

FIGURE A2

ML experiment results for harmonization using only training data. The data is randomly split into 270 training subjects and 235 testing subjects such that every site is represented in each group. The training set is then used to train a random forests algorithm to predict Siemens scanners or patient characteristics. a shows the AUC values for detection of Siemens. AUC values for detection of AD are displayed in b and detection of male in (c). RMSE values for prediction of age are displayed in (d)