Computer vision in animal behaviour : Markerless approaches for fine-scaled behaviour quantification in birds

Abstract

Animals interact with the world through their behaviour. Over the last few decades, technological advancements have provided new ways for animals to be monitored and measured, offering novel insights into the study of animal behaviour. One such advance is the popularization of computer vision methods to measure the position, outline, posture, and behaviour of animals. While these methods have developed at an exponential rate over the last decade, uptake of methods for applications in animal behaviour is still limited, due to various challenges and obstacles that might have prevented computer vision innovations to be directly applied to biological studies. To overcome these challenges and better bridge the two fields, this thesis proposed a generalized framework for computer vision projects in animal behaviour applications, and demonstrated the key opportunities that can facilitate collaboration. Within this framework, the role of the "computer vision practitioner" was emphasized, whose main goal is to implement computer vision pipelines, but also oversees the whole research process. As a demonstration of how this framework can be implemented, this thesis presented two main lines of research. The first is the development of multi-animal, markerless 3D posture estimation methods for fine-scaled behavioural quantification in birds, with the primary aim of measuring head rotation to estimate gaze and attention. To achieve this goal, Chapter 1 presented 3D-POP, a large scale 2D-3D posture dataset in pigeons, and Chapter 2 presented 3D-MuPPET, a framework for multi-animal 3D posture estimation in captivity and the wild. This was the first demonstration of 3D posture estimation for more than 4 individual animals simultaneously, providing a novel framework for fine-scaled, markerless tracking of animal movement and behaviour. The system was extended in Chapter 3, 3D-SOCS, which proposed a synchronized camera system deployed in the wild for 3D posture estimation of great tits. Finally, Chapter 4 proposed a way of evaluating computer vision models based on its performance for the final application using application-specific metrics, through case studies in gaze estimation of pigeons and abundance estimation of chimpanzees. Taken together, the first part of this thesis presented a series of pioneering methods for markerless 3D posture estimation of animals, opening the doors for fine-scaled behavioural tracking across study systems and species. The second part of the thesis aimed to solve the problem of automating behavioural coding in long-term study systems. Chapter 5 presented CHIRP, a task-diverse dataset of Siberian Jays in the wild, supporting action recognition, re-identification, posture estimation, segmentation, object detection, and trajectory tracking. By introducing a novel application-specific benchmark for extracting biologically meaningful measures, the dataset provided a way for computer vision innovations to be directly tested in the context of the final application. Finally, Chapter 6 proposed YOLO-Behaviour, a robust framework for behavioural quantification from videos, demonstrated over 5 different case studies. Collectively, the second part of the thesis laid a foundation for both individual recognition and behavioural annotation from videos, highlighting the value of automating manual annotation, especially in long-term study systems, creating increased sample size for hypothesis testing. Through two lines of research, this thesis proposes that by identifying key synergies and opportunities between the two fields, collaboration between computer vision and biology can be substantially accelerated and improved, with emphasis on the important role of "computer vision practioners". The thesis concluded by formulating these interdisciplinary challenges and opportunities, towards a future where computer vision innovation can contribute to novel data collection procedures across animal systems. With careful collaboration and development, computer vision has the potential to revolutionize the study of animal behaviour, while also transforming fields such as conservation, animal welfare, and beyond.