Gaussian process linking functions for mind, brain, and behavior

Abstract

The link between mind, brain, and behavior has mystified philosophers and scientists for millennia. Recent progress has been made by forming statistical associations between manifest variables of the brain (e.g., electroencephalogram [EEG], functional MRI [fMRI]) and manifest variables of behavior (e.g., response times, accuracy) through hierarchical latent variable models. Within this framework, one can make inferences about the mind in a statistically principled way, such that complex patterns of brain-behavior associations drive the inference procedure. However, previous approaches were limited in the flexibility of the linking function, which has proved prohibitive for understanding the complex dynamics exhibited by the brain. In this article, we propose a data-driven, nonparametric approach that allows complex linking functions to emerge from fitting a hierarchical latent representation of the mind to multivariate, multimodal data. Furthermore, to enforce biological plausibility, we impose both spatial and temporal structure so that the types of realizable system dynamics are constrained. To illustrate the benefits of our approach, we investigate the model's performance in a simulation study and apply it to experimental data. In the simulation study, we verify that the model can be accurately fitted to simulated data, and latent dynamics can be well recovered. In an experimental application, we simultaneously fit the model to fMRI and behavioral data from a continuous motion tracking task. We show that the model accurately recovers both neural and behavioral data and reveals interesting latent cognitive dynamics, the topology of which can be contrasted with several aspects of the experiment.

Keywords: Gaussian process; dimensionality reduction; joint modeling; model-based cognitive neuroscience.

Fig. 1.

Framework for connecting mind, brain, and behavior. Shown is a schematic of the joint modeling framework, where experimental information and structural information specify the structure of a generative model of brain function. The generative model is used to jointly explain all available manifest variables such as response times, blood oxygenated level-dependent response, or electroencephalogram activity. In the present article, a latent Gaussian process is used to link the manifest variables, enabling the most plausible linking function to emerge from fitting the model to data.

Fig. 2.

The structure of Gaussian process joint models. (A) The generative mechanism of the Gaussian process joint models. Given the time vector $\mathbf{t}$, the linking function describing latent cognitive states is modeled by Gaussian processes with a temporal kernel as a function of $\mathbf{t}$ (top right). The linking function, denoted $X$, generates a cognitive state kernel $K\left(X,X\right)$ and is introduced as the covariance function of Gaussian processes modeling neural and behavioral data (bottom left). Depending on the nature of neural data, $K\left(X,X\right)$ can be applied after transformation (e.g., convolution with a hemodynamic response function for fMRI data). In this study, we use automatic relevance determination (ARD) (20) so that each dimension of the latent cognitive state can contribute to the generation of data with different weights of influence. Here, kernel parameters capture sensitivity of data to each latent cognitive dimension using dimension-wise length-scale parameters (bottom right). (B) Details of the ARD-based feature selection mechanism. A small length-scale parameter means that the covariance between two input points is more penalized by their distance and therefore differences in the input values are interpreted as more important (bottom). Meanwhile, a greater length-scale parameter can make even distant input points covary and therefore minor differences in the input are treated as negligible (top).

Fig. 3.

Simulation results. (A) Spatial, latent temporal, and Kronecker-product kernels used to generate synthetic data (Left) and those estimated from fitting the model (Right). Only a subsection of each kernel is presented here for visual clarity. (B) Latent representations used to generate the data (Top) and estimated from fitting the model to data (Bottom). Each representation is shown as a two-dimensional trajectory, arbitrarily color coded according to moments within one of two cycles. (C) Data (black) and model predictions (blue, behavioral; red, neural) are shown for behavioral (Top) and two representative neural (Middle and Bottom) time series. For the model predictions, solid lines are the mean prediction and the surrounding bands are the 95% credible interval.

Fig. 4.

Fits to experiment data. (A) Regions of interest used within the GPJM. One region of interest located in cerebellum is not shown here. (B) Selected time series data from neural (Top and Middle) and behavioral (Bottom) data, along with model fits (neural, red; behavioral, blue), along with 95% credible intervals. (C) A 3D representation of the latent temporal dynamics emerging from the data after fitting the GPJM. Orange dots correspond to gray areas in D, and green, blue, and red lines correspond to projections onto different latent dimensions. (D) Temporal dynamics of the three latent dimensions extracted from the GPJM. Gray highlighted areas in D have corresponding time series information, demarcated by orange dots in C. Note that the latent dynamics in C and D are filtered using the Savitzky–Golay filter with a window size of 21 and the third-order polynomial approximation for visual clarity.

Fig. 5.

Inspection of latent dynamics. (A–C) The same latent dynamics estimated from the data are shown, color coded according to key properties of the data: (A) stimulus coherence (left, red; central, gray; right, green), (B) the position of the joystick (behavioral response: left, orange; central, gray; right, blue), and (C) the result of a clustering analysis on the functional coactivation matrices (clusters 1 to 3: red, yellow, and green, respectively). (D and E) Functional coactivation matrices for different pairwise region of interest combinations. For this analysis, we chose a baseline coactivation cluster, cluster 3 in E, from which to visualize coactivation changes in the remaining clusters. D shows only coactivations whose differences from cluster 3 are larger than a threshold of $\pm$0.2, for visual clarity. SI Appendix provides a complete list of the regions of interest (SI Appendix, Tables S3 and S4), as well as the raw coactivation coefficients (SI Appendix, Fig. S12). Red lines reflect positive correlations whereas blue lines reflect negative correlations. Although the coactivation patterns are consistent between clusters 1 and 3, cluster 1 has generally larger coactivations among the regions of interest. However, cluster 2 involves a strong negative coactivation between the left inferior frontal gyrus (labeled C7) and many other regions of interest.