DeepShapeKit : accurate 4D shape reconstruction of swimming fish

Wu, Ruiheng; Deussen, Oliver; Li, Liang

DeepShapeKit : accurate 4D shape reconstruction of swimming fish

Dateien

Zu diesem Dokument gibt es keine Dateien.

Datum

2022

Publikationstyp

Beitrag zu einem Konferenzband

Publikationsstatus

Published

Erschienen in

MA, Shugen, ed., Yasushi NAKAUCHI, ed. and others. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 23–27, 2022, Kyoto International Conference Center, Japan. Piscataway, NJ: IEEE, 2022, pp. 526-531. ISSN 2153-0858. eISSN 2153-0866. ISBN 978-1-66547-928-8. Available under: doi: 10.1109/IROS47612.2022.9982097

Zusammenfassung

In this paper, we present methods for capturing 4D body shapes of swimming fish with affordable small training datasets and textureless 2D videos. Automated capture of spatiotemporal animal movements and postures is revolutionizing the study of collective animal behavior. 4D (including 3D space + time) shape data from animals like schooling fish contains a rich array of social and non-social information that can be used to shed light on the fundamental mechanisms underlying collective behavior. However, unlike the large datasets used for 4D shape reconstructions of the human body, there are no large amounts of labeled training datasets for reconstructing fish bodies in 4D, due to the difficulty of underwater data collection. We created a template mesh model using 3D scan data from a real fish, then extracted silhouettes (segmentation masks) and key-points of the fish body using Mask R-CNN and DeepLabCut, respectively. Next, using the Adam optimizer, we optimized the 3D template mesh model for each frame by minimizing the difference between the projected 3D model and the detected silhouettes as well as the key-points. Finally, using an LSTM-based smoother, we generated accurate 4D shapes of schooling fish based on the 3D shapes over each frame. Our results show that the method is effective for 4D shape reconstructions of swimming fish, with greater fidelity than other state-of-the-art algorithms.

Fachgebiet (DDC)

004 Informatik

Konferenz

International Conference on Intelligent Robots and Systems (IROS 2022), 23. Okt. 2022 - 27. Okt. 2022, Kyoto, Japan

Zitieren

ISO 690

WU, Ruiheng, Oliver DEUSSEN, Liang LI, 2022. DeepShapeKit : accurate 4D shape reconstruction of swimming fish. International Conference on Intelligent Robots and Systems (IROS 2022). Kyoto, Japan, 23. Okt. 2022 - 27. Okt. 2022. In: MA, Shugen, ed., Yasushi NAKAUCHI, ed. and others. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 23–27, 2022, Kyoto International Conference Center, Japan. Piscataway, NJ: IEEE, 2022, pp. 526-531. ISSN 2153-0858. eISSN 2153-0866. ISBN 978-1-66547-928-8. Available under: doi: 10.1109/IROS47612.2022.9982097

BibTex

@inproceedings{Wu2022DeepS-59666,
  year={2022},
  doi={10.1109/IROS47612.2022.9982097},
  title={DeepShapeKit : accurate 4D shape reconstruction of swimming fish},
  isbn={978-1-66547-928-8},
  issn={2153-0858},
  publisher={IEEE},
  address={Piscataway, NJ},
  booktitle={2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 23–27, 2022, Kyoto International Conference Center, Japan},
  pages={526--531},
  editor={Ma, Shugen and Nakauchi, Yasushi},
  author={Wu, Ruiheng and Deussen, Oliver and Li, Liang}
}

RDF

<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/59666">
    <dc:contributor>Wu, Ruiheng</dc:contributor>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dc:contributor>Deussen, Oliver</dc:contributor>
    <dc:creator>Li, Liang</dc:creator>
    <dc:language>eng</dc:language>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-01-10T13:31:05Z</dcterms:available>
    <dc:rights>terms-of-use</dc:rights>
    <dc:contributor>Li, Liang</dc:contributor>
    <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/>
    <dc:creator>Wu, Ruiheng</dc:creator>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-01-10T13:31:05Z</dc:date>
    <dc:creator>Deussen, Oliver</dc:creator>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/59666"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dcterms:abstract xml:lang="eng">In this paper, we present methods for capturing 4D body shapes of swimming fish with affordable small training datasets and textureless 2D videos. Automated capture of spatiotemporal animal movements and postures is revolutionizing the study of collective animal behavior. 4D (including 3D space + time) shape data from animals like schooling fish contains a rich array of social and non-social information that can be used to shed light on the fundamental mechanisms underlying collective behavior. However, unlike the large datasets used for 4D shape reconstructions of the human body, there are no large amounts of labeled training datasets for reconstructing fish bodies in 4D, due to the difficulty of underwater data collection. We created a template mesh model using 3D scan data from a real fish, then extracted silhouettes (segmentation masks) and key-points of the fish body using Mask R-CNN and DeepLabCut, respectively. Next, using the Adam optimizer, we optimized the 3D template mesh model for each frame by minimizing the difference between the projected 3D model and the detected silhouettes as well as the key-points. Finally, using an LSTM-based smoother, we generated accurate 4D shapes of schooling fish based on the 3D shapes over each frame. Our results show that the method is effective for 4D shape reconstructions of swimming fish, with greater fidelity than other state-of-the-art algorithms.</dcterms:abstract>
    <dcterms:issued>2022</dcterms:issued>
    <dcterms:title>DeepShapeKit : accurate 4D shape reconstruction of swimming fish</dcterms:title>
  </rdf:Description>
</rdf:RDF>

Universitätsbibliographie

Ja