Toward a unified framework for interpreting machine-learning models in neuroimaging

Abstract

Machine learning is a powerful tool for creating computational models relating brain function to behavior, and its use is becoming widespread in neuroscience. However, these models are complex and often hard to interpret, making it difficult to evaluate their neuroscientific validity and contribution to understanding the brain. For neuroimaging-based machine-learning models to be interpretable, they should (i) be comprehensible to humans, (ii) provide useful information about what mental or behavioral constructs are represented in particular brain pathways or regions, and (iii) demonstrate that they are based on relevant neurobiological signal, not artifacts or confounds. In this protocol, we introduce a unified framework that consists of model-, feature- and biology-level assessments to provide complementary results that support the understanding of how and why a model works. Although the framework can be applied to different types of models and data, this protocol provides practical tools and examples of selected analysis methods for a functional MRI dataset and multivariate pattern-based predictive models. A user of the protocol should be familiar with basic programming in MATLAB or Python. This protocol will help build more interpretable neuroimaging-based machine-learning models, contributing to the cumulative understanding of brain mechanisms and brain health. Although the analyses provided here constitute a limited set of tests and take a few hours to days to complete, depending on the size of data and available computational resources, we envision the process of annotating and interpreting models as an open-ended process, involving collaborative efforts across multiple studies and laboratories.

Fig. 1 |. Model complexity in neuroimaging and the model interpretation framework.

a, Neuroimaging-based ML models are usually built upon a large number of features (e.g., ~10⁵ in the case of whole-brain fMRI), which, along with considering potential confounds and correlations between features, makes even linear models complex. In the case of nonlinear models, the situation is more complicated, as it is not clear what a model uses as features. To trust models and find them useful in basic neuroscience and clinical settings, researchers need to know why and how a model works. b, The model interpretation framework consists of three levels of assessment. In the model-level assessment, the model is evaluated as a whole, and the characteristics of the model are derived mainly from observations of the input–output relationship. The assessment includes tests of specificity, sensitivity and generalizability, analyses of model’s representations and decisions and analyses of noise contribution. The feature-level assessment aims to identify features significant for a prediction within a model. The feature significance can be evaluated based on the feature’s impact on predictions or the feature’s stability across multiple samples of the training data. The explanation obtained by this level of assessment should enhance human readability of the model. The biology-level assessment aims to prove the neuroscientific plausibility of the model with evidence from previous literature and other studies using different methodology (e.g., invasive studies). In case a model suggests a novel finding that cannot be verified by the current state of the art, the model can serve as a basis for theory development, which should subsequently be corroborated by studies employing other (e.g., invasive) experimental methods.

Fig. 2 |. A proposed workflow for the procedure.

In this protocol, we present an example workflow that implements the unified model interpretation framework. The first step is the model building (Step 1). This is a prerequisite step that stands outside the interpretation framework. However, we include a brief description of this step to emphasize its importance. This protocol includes examples of building linear support vector machines (option A) and a convolutional neural network model (option B) using an example dataset from our previous publication. Next, we evaluate the basic properties of a model, such as its predictive power (Steps 2 and 3) and contributions of confounds (Steps 4–6). In case either of the two steps shows insufficient quality of the model, one should return to Step 1 to review the data quality and to revise the model. Note that obtaining a definitive answer to the question of Steps 4–6 (i.e., whether the model is confound free) is challenging, and therefore Steps 4–6 should be an open-ended investigation. If the results from Steps 2 and 3 and Steps 4–6 are good enough to move forward, the next step (Step 7) is the feature-level assessment. This protocol provides analysis examples of four options of identifying significant features: bootstrap tests, RFE, ‘virtual lesion’ analysis and LRP. If the identified significant features provide sensible results, one can continue to Steps 8–10 and Step 11. Otherwise (e.g., all the significant features are located within the ventricles), one should revisit the model building. Generalizability testing (Steps 8–10) and biology-level assessment (Step 11) can be performed in an arbitrary order. In Steps 8–10, a model is tested for its generalizability to unseen data from new individuals, different laboratories, scanners and contexts. Testing generalizability requires new test data, which can take a long time to collect. Therefore, one can first examine the model’s biological validity (Step 11) and then test its generalizability, or vice versa. Both generalizability testing and biology-level assessment require open-ended test processes and should support each other; more generalizable models are likely to be more biologically plausible. For Step 11, we provide examples of two options: examining the relationship of the model with the large-scale resting-state functional networks (option A) and term-based meta-analytic decoding using Neurosynth (option B). In practice, this step should also include exhaustive literature reviews and support of invasive studies. The step can also be performed multiple times in case the model suggests novel theories that should be evaluated. The final step of this workflow is the representational analysis (Steps 12–15), which can provide a better understanding of the model’s decision principles by examining the patterns of model behaviors over multiple instances and examples. This step often requires other models with which to be compared, and for this reason, we include this as the last step of the workflow. However, if other models are already available, this step can be done earlier. The results from Steps 12–15 could provide converging evidence for Step 11. Since interpreting an ML neuroimaging model is, in fact, an open-ended process, this workflow should be regarded as the bare minimum, and more analyses other than the ones proposed here can help the model interpretation.

Fig. 3 |. Predictive performance of the SVM model (Steps 2 and 3) and the results of feature-level assessment of the linear models (Step 7, options A and B).

a, The plots illustrate the classification performance of the SVM model tested by LOSO cross-validation with the threshold for misclassification set to 0 (Steps 2 and 3 of the procedure). The top panel shows the cross-validated distance from hyperplane and the decision threshold, and the bottom panel shows the receiver operating characteristic (ROC) plot. The yellow dots indicate correct classification, and the gray dots indicate misclassification. The accuracy of the SVM model reached 92% ± 0.9%. b, The weight map shows significant feature weights of the SVM model identified by the bootstrap tests and thresholded at an FDR of q < 0.05 (Step 7 of the procedure, option A). c, The weight map shows the final predictive SVM features after the RFE procedure, with the final number of features = 20,000 and the number of removed features at each step = 5,000 (Step 7 of the procedure, option B).

Fig. 4 |. A schematic of the ‘virtual lesion’ analysis (Step 7, option C).

The ‘virtual lesion’ analysis investigates how individual regions or networks contribute to final predictions of a model by removing or using one region or network at a time from the model. Based on a selected parcellation, regions in the original model are masked (either one region is removed, or only one region is used for prediction), and the performance of the masked model is evaluated.

Fig. 5 |. Layer-wise relevance propagation results (Step 7, option D).

a and b, We ran the LRP to explain predictions of each trial in each subject and condition (Step 7 of the procedure, option D). We then calculated the average relevance scores across subjects for both conditions. In a, we show the average relevance score map for the heat condition, thresholded at uncorrected P < 0.001, which is equivalent to an FDR at q < 0.029. Similarly, in b, we show the average relevance score map for the rejection condition, thresholded at uncorrected P < 0.001 (equivalent to an FDR at q < 0.019).

Fig. 6 |. Generalizability tests (Steps 8–10).

a, The predictive weight map of the NPS that exceeds an FDR threshold of q < 0.05. b, To assess the generalizability of the NPS (Steps 8–10 of the procedure), we calculated the pattern expression values using the dot product between the signature pattern weights and activation maps for different conditions. Then, we performed the 2AFC test for heat versus warmth and rejection versus friend conditions. The box plot shows the NPS response to the four conditions. The lines between the boxes depict the correct (pink lines) and incorrect (blue lines) classifications. The ROC plot shows the sensitivity and specificity for discriminating between heat and warmth conditions (yellow) and between rejection and friend conditions (blue). c, The predictive weight map of the SIIPS1 that exceed an FDR threshold of q < 0.05. d, The box plot shows the SIIPS1 response to the four conditions. The lines between the boxes depict the correct (pink lines) and incorrect (blue lines) classifications. The ROC plot shows the sensitivity and specificity for discriminating between heat and warmth conditions (yellow) and between rejection and friend conditions (blue).

Fig. 7 |. Examples of biology-level assessment (Step 11) and the representational analysis (Steps 12–15).

a, The radar chart depicts the posterior probability of observing overlaps between the thresholded SVM model and the resting-state functional networks (Step 11, option A). The pink chart represents the overlaps with the positive predictive weights of the model, and the blue chart represents the overlaps with the negative predictive weights of the model. b, The bar plot shows the functional terms obtained from the Neurosynth decoder applied to the unthresholded SVM model (Step 11, option B). The pink bars represent the decoding results for positive weights, and the blue bars represent the decoding results for negative weights. c, In the representational analysis (Steps 12–15), we compared the NPS and SIIPS1 responses to four stimulus conditions. We first compared two accuracy matrices using correlation coefficients and then visualized the relationship between conditions using network and hierarchical clustering methods.