Computer-assisted 3D kinematic analysis of all leg joints in walking insects

Abstract

High-speed video can provide fine-scaled analysis of animal behavior. However, extracting behavioral data from video sequences is a time-consuming, tedious, subjective task. These issues are exacerbated where accurate behavioral descriptions require analysis of multiple points in three dimensions. We describe a new computer program written to assist a user in simultaneously extracting three-dimensional kinematics of multiple points on each of an insect's six legs. Digital video of a walking cockroach was collected in grayscale at 500 fps from two synchronized, calibrated cameras. We improved the legs' visibility by painting white dots on the joints, similar to techniques used for digitizing human motion. Compared to manual digitization of 26 points on the legs over a single, 8-second bout of walking (or 106,496 individual 3D points), our software achieved approximately 90% of the accuracy with 10% of the labor. Our experimental design reduced the complexity of the tracking problem by tethering the insect and allowing it to walk in place on a lightly oiled glass surface, but in principle, the algorithms implemented are extensible to free walking. Our software is free and open-source, written in the free language Python and including a graphical user interface for configuration and control. We encourage collaborative enhancements to make this tool both better and widely utilized.

Figure 1. Experimental setup.

(A) The recording configuration. A cockroach was glued to a flexible tether and walked in place on a plate of oiled glass. One camera was slightly to the front of the animal and the other viewed its ventral surface through a mirror. (B) Ventral view of a cockroach, Blaberus discoidalis, with the body colored black for contrast. The colored dots on the legs indicate the points which were marked and then tracked by our software. The black arrows at right indicate the coordinate system used in this analysis, with the z-axis extending into the page (shaded cube added to indicate depth). The x and y vectors shown would be approximately 1 cm in length. (C) Details of digitized points and definitions of joint angles. (D) Reduction of the trochanter-femur (TrF) joint results in lowering the foot toward the substrate. (E) The thorax-coxa (ThC) joint has three rotational degrees of freedom in the front leg; in the middle and hind legs, only the first two degrees of freedom are actuated.

Figure 2. Image preprocessing.

Leg outlines for the left leg are drawn as an aid to the reader.

Figure 3. Point-extraction algorithm.

(A) Video images have the average background subtracted and are then filtered to amplify and localize the white dots painted on the cockroach's legs. The colored lines approximate the areas shown in greater detail in panel B, though from different images. (B) Proceeding from proximal to distal points on each leg, a 3D ellipsoid is defined based on the last known position of each point, with its short axis along the line between the point and the next-most-proximal point on the same leg. This ellipsoid is projected onto the image from the camera, and the intensity-weighted centroid of the pixels within the resulting ellipse defines the new estimate of that point's 2D position. The blue lines are the ellipsoid axes. The pink ellipses are the 2D projections of the corresponding ellipsoids constraining the search area, and the red Xs show the centroids of the ellipses.

Figure 4. Our user-assisted digitization process yields results with accuracy comparable to manual digitization.

(A) Each colored bar indicates the mean and standard deviation of the 3D Euclidean distance from the manually digitized point to the same point extracted by our software, for a single walking bout (4096 frames). The colored dots on the scale drawings of the legs have a radius corresponding to approximately the average positional error. (B) Each bar shows the mean and standard deviation of the absolute joint angle error. Joint angle errors were distributed log-normally.

Figure 5. Raw 3D positions of the tracked points through a single, 8-second bout of walking.

(A) A ventral view. (B) View from the animal's right and slightly above the substrate. In both panels, the red points indicate the positions of the tibia-tarsus (TiTa) joint; the blue points: the femur-tibia (FTi) joint; green: the coxa-trochanter (CTr) joint; orange: ThC joint; and the purple points are the extra dots placed on the coxae of the front legs to aid in determining their rotation. The black, olive, and yellow line segments connect the points of each leg as they appeared in selected, synchronous video frames. The cockroach did not appear to be walking precisely straight forward during this trial, as most obviously indicated by the left-right asymmetry in the front leg TiTa positions.

Figure 6. Time-series of leg positions during the same bout shown in Fig. 5 .

The black traces show the x (forward-back) positions of the points; red: y (right-left) positions; blue: z (up-down) positions for (A) the TiTa point, (B) the FTi point, and (C) the CTr point.

Figure 7. The measurement and action of the TrF joint.

(A) For one stride by the right middle leg, the motion of the leg is depicted in 3D as a stick figure. Leg segments and planes are colored as in the inset (top left). The plane formed by the ThC-CTr-FTi points is blue, and the plane formed by the CTr-FTi-TiTa points is red. The TrF angle is defined as the angle between these two planes. The black crosses show the projection of the CTr, FTi, and TiTa points onto the ground plane, with a vertical black line connecting each cross to its corresponding joint. The black arrows point toward the animal's head. (B) The FTi, TrF, and CTr joint angles over the same stride plotted in panel A. The 9 colored points correspond to the 9 stick figures in A. (C) High-speed video of one stride by a cockroach walking on an air-floated Styrofoam ball. The lateral edges of the tibia and the coxa were painted with white paint to increase contrast, and are overlaid with red and blue lines here, respectively. The 4 video frames were chosen to approximate the step phases shown by the appropriately numbered points in panels A and B. The red and blue lines are parallel in the top two frames, and rotated in the bottom two frames. This rotation corresponds to reduction at the TrF joint.

Figure 8. Parameters used to calculate stride timing.

(A) Fourier power spectra of the legs' x-positions, for the same walking bout shown in Fig. 5. Green: raw amplitude; black: filtered amplitude. The peak frequency in the black curves is the initial estimate of the stepping rate by each leg. (B) Histograms of the z-positions of the legs' tibia-tarsus (TiTa) joints during the same walking bout. Each histogram is fit with a 2-component Gaussian mixture model (colored traces), with the mean and variance of the lower-valued cluster (pink) used to help determine whether or not the foot was touching the substrate. (C,E,G) Power spectra for other trials, each with a different average step rate (note the different frequencies of the peak power). Panels E and G show the middle legs only. (D,F,H) Z-histograms for the same bouts.

Figure 9. Calculation of stride timing.

(A) Enlarged view of a portion of the walking bout shown in Fig. 5. Although it is not obvious, the x-, y-, and z-positions of the TiTa points (shown in black, red, and blue, respectively) are displayed here as points rather than lines to demonstrate the discretization due to the digital video capture. Discontinuities typically occurred when the forward and backward tracking intersected (see Methods). The gray boxes denote the stance phases of each leg, as defined by the leg's z-coordinate. The dashed, green boxes show how the stance phase calculation changes when the foot's anterior and posterior extreme positions (AEP and PEP, respectively) are used to determine stride timing.

Figure 10. Joint angles calculated from 3D joint positions during a portion of the same walking bout shown in Fig. 5 .

As in Fig. 8, the colored points represent successive frames of tracked video. The gray boxes indicate the stance phase of each stride, defined using the z-position of the TiTa point. The FTi joint angle (blue) was estimated using the CTr, FTi, and TiTa points. The TrF joint angle (pink) was estimated using the CTr-FTi-TiTa plane and the coxal plane. The CTr angle (black) was estimated using the ThC, CTr, and FTi points. The angle of the coxa in the xz-plane centered at the ThC position was used to define the motion in the first degree of freedom (promotion/remotion) of the ThC joint (ThC₁, green), and the declination of the coxa from the xy-plane determined its motion in the ThC joint's second degree of freedom (ThC₂, orange).

Figure 11. The duty cycle of the feet changes with walking speed.

(A) 2D histogram of the duty cycle for each stride, plotted versus that stride's duration. Warmer colors indicate more strides in a given bin (total 763 strides from 25 bouts of walking by 8 animals). The duty cycle indicates the fraction of each stride during which the foot was on the ground, and generally decreases with increasing speed (or decreasing period). Here, the stance phase was calculated by AEP and PEP, as for the green boxes in Fig. 8. (B) Duty cycle, pooled by trial. Each diamond represents the mean and s.e.m. of both stride period and duty cycle for a single bout of walking. (C) Same data as panel A, but with stance phase determined by the TiTa points' z-values, as for the gray boxes in Fig. 8. (D) Same as panel B, but using z-values to delineate strides.