Long-duration animal tracking in difficult lighting conditions

Abstract

High-throughput analysis of animal behavior requires software to analyze videos. Such software typically depends on the experiments' being performed in good lighting conditions, but this ideal is difficult or impossible to achieve for certain classes of experiments. Here, we describe techniques that allow long-duration positional tracking in difficult lighting conditions with strong shadows or recurring "on"/"off" changes in lighting. The latter condition will likely become increasingly common, e.g., for Drosophila due to the advent of red-shifted channel rhodopsins. The techniques enabled tracking with good accuracy in three types of experiments with difficult lighting conditions in our lab. Our technique handling shadows relies on single-animal tracking and on shadows' and flies' being accurately distinguishable by distance to the center of the arena (or a similar geometric rule); the other techniques should be broadly applicable. We implemented the techniques as extensions of the widely-used tracking software Ctrax; however, they are relatively simple, not specific to Drosophila, and could be added to other trackers as well.

Figure 1. Sample of the difficult lighting conditions our techniques address.

(a) Schematic of the cross section of our egg-laying chamber. We recorded through the lid with a camera above the chamber. Grape juice provided food for the flies. (b) Sample video frame from our “UV on” experiments, showing two chambers. For the left chamber, the top edge of the chamber sidewall is outlined in yellow, and the two egg-laying sites at the bottom of the chamber are outlined in white. The blue arrow points to a UV LED. Note that there is one fly per chamber (white arrows). A red “light pad” under the chambers provides additional lighting – that is invisible to Drosophila – for tracking. (c) Sample frame from our “UV on/off” experiments at a time when UV is off. The small dark spots on the egg-laying sites are eggs (arrows). (d-g) Sample frame with strong shadows that led to false positives, with white and green arrows pointing to the two flies. (d) Frame in grayscale, which Ctrax uses for tracking. (e) Difference between frame (d) and background (see text), with darkness proportional to the absolute value of the difference. The shadows (blue arrows) of the right fly have a larger difference than the fly itself. (We used Ctrax’s “Background Brightness” normalization, which performed best for our chambers.) (f) The same difference as in (e) now shown in green and superimposed onto the background. (g) Flies detected by Ctrax shown as ellipses. The ellipses without arrow are false positives.

Figure 2. Techniques used for our “UV on” experiments.

(a) Sample frame (same as Fig. 1g) illustrating shadow detector, which (correctly) picks the flies pointed to by white arrows. See text for details. (b-c) Template matching. See text for details. (b) Template image (showing bottom of chambers in good lighting conditions). (c) Background image with white rectangle indicating where the template best matches the background. (d-e) Detecting tracking errors. (d) 1-hour trajectory with tracking error. The blue and green arrows point to multiple jumps (straight lines) to or from the center of the chamber (yellow arrow), with the blue arrow pointing to multiple almost identical jumps. (e) Sample jump reported by our suspicious jump detector. The image shows three consecutive partial (left chamber only) frames (i, i + 1, i + 2), with the ellipses showing where Ctrax reports the fly is and the arrow pointing to the fly’s actual position in the error frame (i + 1). (f) Background recalculation for an 8 h “UV on” video. The background calculated over the 1st hour is shown at the top of the panel (bg1). Differences between bg1 and the backgrounds calculated over the 2nd and 8th hours (Δbg2, bg1 and Δbg8, bg1) are shown below bg1. Some of the background changes here (arrows) are caused by changes in grape juice level in the center well. (g) Typical 1-hour trajectories of two (not-too-active) flies using Ctrax with extensions on an 8h “UV on” video with strong shadows. Arrows point to shorter incorrect jumps (easiest to see for blue arrow).

Figure 3. Additional techniques used for our “UV on/off” experiments.

(a) Backgrounds calculated by the on/off detector for an 8h “UV on/off” video. Separate backgrounds for “on” and “off” were calculated over each hour, with the backgrounds for the 1st hour shown at the top of the panel (off1, on1). Differences between off1 and the backgrounds for “off” calculated over the 2nd and 8th hours (Δoff2,off1 and Δoff8,off1) are shown below off1. Corresponding differences are shown below on1. Note that eggs laid over the UV LEDs caused large differences when the LEDs were on (green arrows), including strong shadows on the chamber sidewalls (blue arrows). (b-c) Tracking error fixed by the suspicious jump detector, with the images showing three consecutive partial (left chamber only) frames (j, j + 1, j + 2) both before (b) and after (c) the fix. The ellipses show where Ctrax reported the fly was and the white arrow points to the actual position of the fly in the error frame. Strictly speaking, not Ctrax but the suspicious jump detector reported the positions in (c); the detector is MATLAB code separate from Ctrax. Note that the LED in frame j + 1 is not fully on (compare, e.g., the area of the LED pointed to by the blue arrows in frames j + 1 and j + 2), which caused the suspicious jump.

Figure 4. UV attraction of Drosophila virgins in “UV on/off” experiments.

(a) Chamber for “UV on/off” experiments at a time when UV is on. The green lines indicate the borders of the top and bottom sites. (b) Plot of y position of a fly in the chamber over 1 minute. A “bottom return” occurs when the fly leaves the bottom site and returns to it without reaching the top site in-between (the brown square marks the point where the fly had gotten the closest to the top site). A “top return” is defined correspondingly. (c) Plot of y position of a fly over 20 minutes when the UV on/off state changes every 4 minutes. (d) Return preference index (PI) during UV off and UV on times when the on/off state changes every 4 minutes. 12 wild type virgins were recorded for 3 hours each. After tracking, for each hour of the trajectories, we determined the numbers of bottom and top returns for UV off and UV on separately and calculated the return PIs for UV off and UV on as (number of bottom returns - number of top returns) / (total number of returns). n = 34 and 36 hours per bar (two of the hours had no returns for UV off), bars show mean with SEM, unpaired t-test with Welch’s correction, p < 0.0001, two-tailed. (e) Total number of returns by hour during UV off and UV on times when the on/off state changes every 4 minutes. n = 36 hours per bar, Mann-Whitney test, p < 0.0001, two-tailed.