Deep attention networks reveal the rules of collective motion in zebrafish

Abstract

A variety of simple models has been proposed to understand the collective motion of animals. These models can be insightful but may lack important elements necessary to predict the motion of each individual in the collective. Adding more detail increases predictability but can make models too complex to be insightful. Here we report that deep attention networks can obtain a model of collective behavior that is simultaneously predictive and insightful thanks to an organization in modules. When using simulated trajectories, the model recovers the ground-truth interaction rule used to generate them, as well as the number of interacting neighbours. For experimental trajectories of large groups of 60-100 zebrafish, Danio rerio, the model obtains that interactions between pairs can approximately be described as repulsive, attractive or as alignment, but only when moving slowly. At high velocities, interactions correspond only to alignment or alignment mixed with repulsion at close distances. The model also shows that each zebrafish decides where to move by aggregating information from the group as a weighted average over neighbours. Weights are higher for neighbours that are close, in a collision path or moving faster in frontal and lateral locations. The network also extracts that the number of interacting individuals is dynamical and typically in the range 8-22, with 1-10 more important ones. Our results suggest that each animal decides by dynamically selecting information from the collective.

Fig 1. Deep-learning a model of collective behaviour.

(A) Variables used to predict future turns. Asocial variables, those only involving the focal, in red. Social variables, those involving both the focal and a neighbour, in blue. (B) Pair-interaction subnetwork, receiving asocial variables α and social variables σ_i from a single neighbour i, and outputting a vector of 128 components. All pair-interaction networks share the same weights. (C) Interaction network, showing how the outputs of the pair-interaction subnetworks, one for each neighbour, are summed and then fed to an interaction subnetwork. The output, z is the logit of the focal fish turning right after 1 s. (D) Pair-interaction subnetwork of the attention network. (E) Aggregation subnetwork of the attention network. Same structure as D, but the input is a restricted symmetric subset of the variables and the output is passed through an exponential function to make it positive. (F) Attention network, showing how the inputs of the pair-interaction and aggregation subnetworks are integrated to produce a single logit z for the focal fish turning right after 1 s.

Fig 2. Properties of interaction between a pair of fish in the collective.

(A) Logit z resulting from the pair-interaction subnetwork of the attention network, plotted as a function of the orientation of the neighbour respect to the focal, θ_i, and speed of the neighbour, v_i, for neighbour located at (x_i, y_i) = (7, 1) BL (body lengths, left) and (x_i, y_i) = (3, 1) BL (right). Focal speed is fixed at median speed of 3.04 BL/s and focal acceleration at a_⊥ = 0 BL/s². Red colour is evidence that the focal fish will turn right in 1s, while blue is evidence that the focal fish will turn left. Horizontal dashed line highlights the median speed of 3.04 BL/s. (B) Same as (A) but for 64 different neighbour positions (x_i, y_i), with x_i and y_i taking values in (−7, −5, −3, −1, 1, 3, 5, 7) BL.

Fig 3. Alignment, attraction and repulsion areas depend on kinematic parameters of focal and neighbour.

(A, B). Alignment (gray), attraction (orange), repulsion (purple) and anti-alignment (pink) areas. Alignment score (gray) measures how much the logit changes when changing the neighbour orientation angle, and it is computed only in the orientation areas (see Methods and materials). Attraction (orange) and repulsion scores (purple) are the logit averaged across relative orientation angles (positive or negative, respectively), plotted outside orientation areas. (A) Scores given at four different values of the neighbour speed (1, 2, 4 and 8 BL/s) while fixing focal speed at the median 3.04 BL/s. Focal normal acceleration fixed at a_⊥ = 0. (B) Same as (A) but now fixing neighbour speed and varying focal speed. (C, D) Attraction and repulsion scores as in (A,B) but now plotted for all regions regardless of whether there is alignment effect or not.

Fig 4. Ground-truth validation using simulated trajectories with known interaction rules.

(A) Interaction models used to generate data. Fish turn away from neighbours that are in the repulsion area. If there are no neighbours in the repulsion area, fish align with neighbours in the alignment area and are attracted to neighbours in the attraction area. The shape, size and relative location of the areas is varied in different simulations, shown here in different columns. (B) Pair interaction scores (above) and aggregation weights (below) obtained when training using simulated trajectories generated by the interaction rules in A. (C) Average total number of interacting neighbours, N_total, (blue) as estimated from the deep attention network. Each dot corresponds to a different video with different maximum number of interacting individuals of 3, 7, 11, 15, 19 and 23. Exact correspondence with ground truth number of interacting neighbours as black line. Right: histogram (upper) and example time series (lower) for the estimated number of interacting neighbours, N_total, (blue) and the groundtruth (black). Histogram and time series calculated for trajectories where a maximum of 15 individuals interact.

Fig 5. Weighting function: How a fish aggregates information from neighbours.

Logarithm of the aggregation weight, log(W), as a function of neighbour position, x_i and y_i. Top row: focal speed fixed at 3.04 BL/s and each subplot corresponding to different neighbour speeds marked on top of each. Bottom row: same as top row but for fixed neighbour speed at 3.04 BL/s and different focal speeds.

Fig 6. Relevant neighbours in the aggregation.

A (Upper panel) Three example frames with each neighbour coloured with its normalized weight in the aggregation. Focal animal is indicated in gray color, with a horizontal line proportional to the normal acceleration to either left or right and a small dot in its frontal positions indicating the focal position 1 second into the future. In both focal and neighbours, a line along the major axis indicates the fish velocity. (Lower panel) Distribution of the total number of interacting neighbours N_total (blue), and of the important neighbours, N_important (red). B Left: Time variation of the number of total interacting neighbours, N_total, and the number of important neighbours, N_important for a illustrative focal fish and short period of time. Right: Power spectra of the two measures of the estimated number of neighbours.