RoboTwin : A Platform to Study Hydrodynamic Interactions in Schooling Fish
2024, Li, Liang, Chao, Li-Ming, Wang, Siyuan, Deussen, Oliver, Couzin, Iain D.
By living and moving in groups, fish can gain many benefits, such as heightened predator detection, greater hunting efficiency, more accurate environmental sensing, and energy saving. Although the benefits of hydrodynamic interactions in schooling fish have drawn growing interest in fields such as biology, physics, and engineering, and multiple hypotheses for how such benefits may arise have been proposed, it is still largely unknown which mechanisms fish employ to obtain hydrodynamic benefits, such as increased thrust or improved movement efficiency. One main bottleneck has been the difficulty in collecting detailed sensory information, corresponding locomotory responses, and hydrodynamic information from real schooling fish.
DeepShapeKit : accurate 4D shape reconstruction of swimming fish
2022, Wu, Ruiheng, Deussen, Oliver, Li, Liang
In this paper, we present methods for capturing 4D body shapes of swimming fish with affordable small training datasets and textureless 2D videos. Automated capture of spatiotemporal animal movements and postures is revolutionizing the study of collective animal behavior. 4D (including 3D space + time) shape data from animals like schooling fish contains a rich array of social and non-social information that can be used to shed light on the fundamental mechanisms underlying collective behavior. However, unlike the large datasets used for 4D shape reconstructions of the human body, there are no large amounts of labeled training datasets for reconstructing fish bodies in 4D, due to the difficulty of underwater data collection. We created a template mesh model using 3D scan data from a real fish, then extracted silhouettes (segmentation masks) and key-points of the fish body using Mask R-CNN and DeepLabCut, respectively. Next, using the Adam optimizer, we optimized the 3D template mesh model for each frame by minimizing the difference between the projected 3D model and the detected silhouettes as well as the key-points. Finally, using an LSTM-based smoother, we generated accurate 4D shapes of schooling fish based on the 3D shapes over each frame. Our results show that the method is effective for 4D shape reconstructions of swimming fish, with greater fidelity than other state-of-the-art algorithms.