Identifying behavioral structure from deep variational embeddings of animal motion

Abstract

Quantification and detection of the hierarchical organization of behavior is a major challenge in neuroscience. Recent advances in markerless pose estimation enable the visualization of high-dimensional spatiotemporal behavioral dynamics of animal motion. However, robust and reliable technical approaches are needed to uncover underlying structure in these data and to segment behavior into discrete hierarchically organized motifs. Here, we present an unsupervised probabilistic deep learning framework that identifies behavioral structure from deep variational embeddings of animal motion (VAME). By using a mouse model of beta amyloidosis as a use case, we show that VAME not only identifies discrete behavioral motifs, but also captures a hierarchical representation of the motif's usage. The approach allows for the grouping of motifs into communities and the detection of differences in community-specific motif usage of individual mouse cohorts that were undetectable by human visual observation. Thus, we present a robust approach for the segmentation of animal motion that is applicable to a wide range of experimental setups, models and conditions without requiring supervised or a-priori human interference.

Fig. 1. VAME: an unsupervised deep learning model for behavior segmentation.

a VAME workflow. Data acquisition via bottom-up camera setup for precise body and limb kinematics. Pose estimation of bottom view (DeepLabCut). Frames are egocentrically aligned and trajectory samples are fed into the recurrent neural network model. The fully trained model resembles a dynamical system from which motifs are inferred via a Hidden-Markov-Model. b Example trace of an egocentric aligned DLC time series showing a full walking cycle (phase block). The corresponding motif sequence segmented by VAME has repeated motifs during the phase block cycle. The matching video frames identify the phase block as a full walking cycle performed by the animal.

Fig. 2. Behavioral quantification with VAME and hierarchical community clustering.

a Locomotor activity of transgenic (tg, n = 4) and wildtype (wt, n = 4) animals. b Hierarchical representation of behavioral motifs. Color grouping on the tree are depicting communities of motifs which belong to the same observable category of behavior. A depiction of up- and downregulation of motifs and communities in tg animals (red line) compared to wt animals (green line) and ordered by communities is shown below. c Quantification of motif usage in percent (%) ordered by communities. Differences between the tg and wt phenotype are in community b, c, e and g. d Visual representation of significantly changes motifs. The start frame is colored in cyan and the end frame is colored in magenta. White dots represent the DLC virtual marker points. e Time-dependent modulation of significantly changed behavioral motifs for both phenotypes binned into six blocks. Error bars represent standard deviation.

Fig. 3. Identification of transition structure and locomotion patterns.

a Transition probability matrices ordered by communities for the wt and tg group and the corresponding difference plot of both matrices. Squares along the diagonal indicate the grouping into communities. b Example of an intra-community transition graph for the walking community. The first two graphs are showing the two highest transitions for both groups and the third graph shows the highest transition difference. c Joint UMAP embedding of points belonging to the walking community in a wt (19.783 points, black) and a tg (13.264 points, red) mouse reveals a circular structure. The projection of the mean phase angle of the horizontal hind paw movement onto the embedding displays the cyclic phase space of the walking movement in both animals. Parametrization of both point clouds with k-means shows blocks organized around the cyclic structure. Red arrow indicates the phase direction.

Fig. 4. Annotated dataset and model comparison based on annotator agreement.

a Overlap of manually assigned labels by three experts. b Disagreement in manual annotation. c Confusion matrices showing the annotator variability (blue) and the agreement between 50 model (VAME, AR-HMM, MotionMapper) motifs and 5 manually annotated labels (red). Empty columns exist when the specific motif did not appear in the annotated benchmark data. d Model evaluation on three metrics (Purity, NMI, Homogeneity) as a function of number of motifs k.

Fig. 5. Workflow figure for VAME and the corresponding code functions.

The figure shows the complete life cycle of a VAME project. The main steps are the project initialization, the transformation of the data from the pose .csv file to a Python .npy file, the creation of a training and test dataset, to training and evaluating the model, and lastly to segment the pose data into behavioral motifs. Afterwards, user can invest behavioral motifs by creating videos from these episodes.

Fig. 6. VAME project folder structue for one video (video-1.mp4).

By initializing a VAME project a project folder will be created within the working directory. This folder contains the raw data, the configuration file (config.yaml), the trained VAME model(s), and the results (segmentation, latent vectors, motif videos).