DeepLabStream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection

Abstract

In general, animal behavior can be described as the neuronal-driven sequence of reoccurring postures through time. Most of the available current technologies focus on offline pose estimation with high spatiotemporal resolution. However, to correlate behavior with neuronal activity it is often necessary to detect and react online to behavioral expressions. Here we present DeepLabStream, a versatile closed-loop tool providing real-time pose estimation to deliver posture dependent stimulations. DeepLabStream has a temporal resolution in the millisecond range, can utilize different input, as well as output devices and can be tailored to multiple experimental designs. We employ DeepLabStream to semi-autonomously run a second-order olfactory conditioning task with freely moving mice and optogenetically label neuronal ensembles active during specific head directions.

Fig. 1. A visual representation of DLStream.

Visual representation of workflow in DLStream. Initially, an experimental protocol is designed using a sequence of modules (puzzle pieces) and a trained DLC network is integrated into DLStream. Afterward, DLStream provides three different outputs for every experiment. 1. Experiments can be monitored on a live stream. 2. The experimental protocol is run based on posture detection 3. Recorded video and experimental data are exported after the experiment is done.

Fig. 2. Experimental design using DLStream.

a, b Schematic design of an experimental protocol with a posture-based trigger. Manipulation can be turned “Conditional OFF” (a) and “Conditional ON” (b) based on the mouse’s behavior. The combination of several modules allows building a sophisticated experimental protocol. For example, the timer module can be utilized to design inter-trial and -stimulus timers (b), minimum stimulation (b), or delayed triggers (e). c Description of available modules in a and b. d Application of the above-described design in an optogenetic experiment. The stimulation is triggered dependent on head direction angle (orange arrow, α) to a reference point (red line) within the target window (blue arc). e Application of the above-described design in a classical conditioning task. The mouse is shown an image when looking at the screen (left) and the reward is removed if it does not move into the reward location within a predefined timeframe (right, green zone). The mouse’s posture is shown with orange dots.

Fig. 3. Closed-loop conditioning task.

a Conditioning. When a trial is triggered by the mouse facing the screen (green triangle and ring), the mouse is shown a visual stimulus (yellow lightning bolt). Mice not facing the screen do not receive the stimulus (red x). In the positive trial (green lightning bolt, green line), a reward is delivered (blue drop, arrow down) and withdrawn (blue drop, arrow up) if not collected within a preset time period. In the negative trial (blue lightning bolt, blue line) only a loud tone (red polygon) is delivered. b 2nd Order conditioning. Upon exploration of either odor location (colored black circle) the mouse is shown one of the previously conditioned visual stimuli on the screen (yellow lightning bolt). Conditioning was conducted in two stages. The first stage (Stage 1) consisted of direct contact with the odor location, while the second (Stage 2) was dependent on the proximity of the mouse to one of the locations (black arrow) and the mouse facing towards it. c Odor preference task. The mouse was set in an open field arena with one odor in each of the quarters (colored circles). The total investigation time of each odor source was measured. d Investigation time during odor preference task in odor location: ROIs encircling the odor location. The bar graph shows the STD and individual data points. p < 0.05 (*) one-tailed paired t-test; R/V p = 0.0395, R/VA p = 0.0497, R/A p = 0.0311; n = 11 trials (2 trials per mouse, 1 trial excluded, 6 mice total; see also Supplementary Data 1). Error bars represent standard deviation. V = Vanillin (S+), R = Rose (S−), VA = Valeric acid, A = Acetophenone.

Fig. 4. Optogenetic labeling of head direction-dependent neuronal activity.

a Left: Stereotactic delivery of Cal-Light viruses into the ADN and fiber ferrule placement. Middle: Infected neurons (red) are stimulated with blue light (488 nm) controlled by DLStream. Right: Infected neurons are only labeled (yellow) when they are active (black arrow) during light stimulation (middle). b Schematic drawing of the circular arena with the visual cue (thick black arc) and the target window (thick blue arc) around the reference point (red circle). DLStream triggered stimulation is strictly dependent on the correct head direction (blue arc). c Left: Representative example (see also Supplementary Data 2) radial histogram of all head directions during stimulation (red) within one session (normalized to the maximum value). The mean resultant vector length is indicated by r. Right: Radial histogram of all head directions during the whole session (gray) and during stimulation (red) (normalized to the maximum value of the entire session). Rings represent quantiles in 20% steps. d Left: Representative random sample of the whole session simulating stimulation without DLStream control at random time points during the session (normalized to the maximum value). The mean resultant vector length is indicated by r. For each session, random distributions were calculated 1000 times. Right: For one session, the distribution of mean resultant vector lengths generated by random sampling (n = 1000). The red line denotes the actual mean resultant vector length during stimulation in the session. The dotted black line represents the p < 0.01 cutoff. e Representative example of the mouse’s position (gray) over time during the first 5 min of the session in c. The stimulation events are shown in blue. f Heatmaps representing the relative occupancy of the mouse within the arena during the whole session (top) and stimulation (bottom) in c. Cue and target window are shown in their relative position. g Example of Cal-Light expression in an experimental mouse. Left: tdTomato expression (red) indicating expression of Cal-Light viruses with nucleus staining (DAPI, blue). Right: Activity-dependent and light-induced eGFP expression (green). The white box represents the zoomed-in region in h. The bar represents 200 µm. h Close up from g vs. a similar region in an animal that was not stimulated with light (no light). Left: tdTomato expression (red). Right: Activity dependent and light-induced eGFP expression (green). The bar represents 50 µm. Note that control mice show no eGFP expression. i Average light stimulation during each session (40 total) corresponding to head direction (60° bins) with target window (blue) indicating the DLStream triggered stimulation onset (see also Supplementary Data 3). Paired student’s t-test: p < 0.001. n = 10 mice. Error bars represent standard deviation. j Ratio between infected neurons (tdTom⁺) and activity-dependent labeled neurons (eGFP⁺/tdTom⁺) in mice matching selection criteria (see “Methods” section). n = 2 mice.