Using computer vision to study animal behavior in natural environments

Abstract

Understanding how animals behave in the context of their physical and social environments is fundamental to the study of animal behavior. While observing animal behavior in the field is always challenging, it becomes much harder, and in some cases even impossible when multiple animals are behaving together. Despite this challenge, collective dynamics are of wide interest across fields ranging from robotics to neuroscience to conservation. While catching and tagging animals is a common way to study animal movement dynamics in complex environments, this approach has many drawbacks. As an alternative, I highlight the utility of pairing high-resolution imaging with convolutional neural networks to study behavior in the field. In this thesis, I present three chapters, each of which presents a new method that allows researchers to extract high quality data about groups of animals behaving in naturalistic environments.In chapter 1, I demonstrate a system that, using ten inexpensive consumer cameras, can robustly estimate the population size of the largest fruit bat colony in Africa. Beyond population estimates, this method also quantifies behavioral measures like flight altitude, wing beat frequency and local neighborhood polarity for thousands of bats each night. I combine threshold-based object detection techniques with neural networks to achieve the robustness of neural networks but without humans having to explicitly annotate training data.In chapter 2, I demonstrate a system that combines imaging drones, machine learning, and landscape mapping techniques to record the movement of free ranging groups of animals at high spatial and temporal resolution across natural landscapes. This approach reconstructs the trajectories of all individuals in the group in 3D landscape maps generated from the same 4 observation videos while also quantifying individual features like animal posture, species, and age/sex class.Finally, in chapter 3, I demonstrate a system that uses semantic segmentation to classify, for every pixel in an image of a honey bee frame, both its comb cell type as well as any contents that it may contain (honey or pollen for example). For the first time, this purely automated method allows researchers to study, through data about the nest, how honey bees, a model organism for distributed organization, build their nests and allocate space within it. Combined, these three chapters clearly demonstrate the potential for using automated image-based approaches to study animal behavior in complex natural environments.