Automated pose estimation in primates

Abstract

Understanding the behavior of primates is important for primatology, for psychology, and for biology more broadly. It is also important for biomedicine, where primates are an important model organism, and whose behavior is often an important variable of interest. Our ability to rigorously quantify behavior has, however, long been limited. On one hand, we can rigorously quantify low-information measures like preference, looking time, and reaction time; on the other, we can use more gestalt measures like behavioral categories tracked via ethogram, but at high cost and with high variability. Recent technological advances have led to a major revolution in behavioral measurement that offers affordable and scalable rigor. Specifically, digital video cameras and automated pose tracking software can provide measures of full-body position (i.e., pose) of primates over time (i.e., behavior) with high spatial and temporal resolution. Pose-tracking technology in turn can be used to infer behavioral states, such as eating, sleeping, and mating. We call this technological approach behavioral imaging. In this review, we situate the behavioral imaging revolution in the history of the study of behavior, argue for investment in and development of analytical and research techniques that can profit from the advent of the era of big behavior, and propose that primate centers and zoos will take on a more central role in relevant fields of research than they have in the past.

Keywords: behavioral tracking; big data; deep learning; primates; rhesus macaque.

Figure 1.

Camera tracking system at the Minnesota Zoo. Relatively affordable cameras, such as GoPro cameras, when combined with computer vision tracking software, can provide estimates of pose, although they depend on having long-term access and a safe vantage point.

Figure 2.

Automated pose-tracking software, such as OpenMonkeyStudio (Bala et al., 2020), can provide high quality tracking of poses in primates. The OMS system is based on multiview capture, which can bypass problems associated with occlusion.

Figure 3.

With OpenMonkeyStudio, we compare 2D and 3D representations in macaques (Bala et al., 2020). We compared our systems’ ability to recognize semantic actions (standing, walking, climbing, climbing supine, sitting, and jumping). A. The poses are clustered by using UMAP. Each cluster is correlated with specific actions. B. With the clusters, we recognize actions using the k nearest neighbor search and visualize the transitions between actions. C. In contrast, the 2D representation provides the clusters that are driven by the pose and viewpoint.

Figure 4.

Images from our new dataset, OpenMonkeyChallenge, illustrating the range of species, individuals, and backgrounds.

Figure 5.

The orangutan enclosure at the Toronto Zoo allows for panoramic viewing of the research subjects, who have a large and enriched space to move in. Such enclosures are particularly useful for behavioral imaging.