DeepShapeKit : accurate 4D shape reconstruction of swimming fish

No Thumbnail Available
Files
There are no files associated with this item.
Date
2022
Editors
Contact
Journal ISSN
Electronic ISSN
ISBN
Bibliographical data
Publisher
Series
URI (citable link)
ArXiv-ID
International patent number
Link to the license
EU project number
Project
Open Access publication
Restricted until
Title in another language
Research Projects
Organizational Units
Journal Issue
Publication type
Contribution to a conference collection
Publication status
Published
Published in
2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 23–27, 2022, Kyoto International Conference Center, Japan / Ma, Shugen; Nakauchi, Yasushi et al. (ed.). - Piscataway, NJ : IEEE, 2022. - pp. 526-531. - ISSN 2153-0858. - eISSN 2153-0866. - ISBN 978-1-66547-928-8
Abstract
In this paper, we present methods for capturing 4D body shapes of swimming fish with affordable small training datasets and textureless 2D videos. Automated capture of spatiotemporal animal movements and postures is revolutionizing the study of collective animal behavior. 4D (including 3D space + time) shape data from animals like schooling fish contains a rich array of social and non-social information that can be used to shed light on the fundamental mechanisms underlying collective behavior. However, unlike the large datasets used for 4D shape reconstructions of the human body, there are no large amounts of labeled training datasets for reconstructing fish bodies in 4D, due to the difficulty of underwater data collection. We created a template mesh model using 3D scan data from a real fish, then extracted silhouettes (segmentation masks) and key-points of the fish body using Mask R-CNN and DeepLabCut, respectively. Next, using the Adam optimizer, we optimized the 3D template mesh model for each frame by minimizing the difference between the projected 3D model and the detected silhouettes as well as the key-points. Finally, using an LSTM-based smoother, we generated accurate 4D shapes of schooling fish based on the 3D shapes over each frame. Our results show that the method is effective for 4D shape reconstructions of swimming fish, with greater fidelity than other state-of-the-art algorithms.
Summary in another language
Subject (DDC)
004 Computer Science
Keywords
Conference
International Conference on Intelligent Robots and Systems (IROS 2022), Oct 23, 2022 - Oct 27, 2022, Kyoto, Japan
Review
undefined / . - undefined, undefined. - (undefined; undefined)
Cite This
ISO 690WU, Ruiheng, Oliver DEUSSEN, Liang LI, 2022. DeepShapeKit : accurate 4D shape reconstruction of swimming fish. International Conference on Intelligent Robots and Systems (IROS 2022). Kyoto, Japan, Oct 23, 2022 - Oct 27, 2022. In: MA, Shugen, ed., Yasushi NAKAUCHI, ed. and others. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 23–27, 2022, Kyoto International Conference Center, Japan. Piscataway, NJ:IEEE, pp. 526-531. ISSN 2153-0858. eISSN 2153-0866. ISBN 978-1-66547-928-8. Available under: doi: 10.1109/IROS47612.2022.9982097
BibTex
@inproceedings{Wu2022DeepS-59666,
  year={2022},
  doi={10.1109/IROS47612.2022.9982097},
  title={DeepShapeKit : accurate 4D shape reconstruction of swimming fish},
  isbn={978-1-66547-928-8},
  issn={2153-0858},
  publisher={IEEE},
  address={Piscataway, NJ},
  booktitle={2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 23–27, 2022, Kyoto International Conference Center, Japan},
  pages={526--531},
  editor={Ma, Shugen and Nakauchi, Yasushi},
  author={Wu, Ruiheng and Deussen, Oliver and Li, Liang}
}
RDF
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/59666">
    <dc:contributor>Wu, Ruiheng</dc:contributor>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dc:contributor>Deussen, Oliver</dc:contributor>
    <dc:creator>Li, Liang</dc:creator>
    <dc:language>eng</dc:language>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-01-10T13:31:05Z</dcterms:available>
    <dc:rights>terms-of-use</dc:rights>
    <dc:contributor>Li, Liang</dc:contributor>
    <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/>
    <dc:creator>Wu, Ruiheng</dc:creator>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-01-10T13:31:05Z</dc:date>
    <dc:creator>Deussen, Oliver</dc:creator>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/59666"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dcterms:abstract xml:lang="eng">In this paper, we present methods for capturing 4D body shapes of swimming fish with affordable small training datasets and textureless 2D videos. Automated capture of spatiotemporal animal movements and postures is revolutionizing the study of collective animal behavior. 4D (including 3D space + time) shape data from animals like schooling fish contains a rich array of social and non-social information that can be used to shed light on the fundamental mechanisms underlying collective behavior. However, unlike the large datasets used for 4D shape reconstructions of the human body, there are no large amounts of labeled training datasets for reconstructing fish bodies in 4D, due to the difficulty of underwater data collection. We created a template mesh model using 3D scan data from a real fish, then extracted silhouettes (segmentation masks) and key-points of the fish body using Mask R-CNN and DeepLabCut, respectively. Next, using the Adam optimizer, we optimized the 3D template mesh model for each frame by minimizing the difference between the projected 3D model and the detected silhouettes as well as the key-points. Finally, using an LSTM-based smoother, we generated accurate 4D shapes of schooling fish based on the 3D shapes over each frame. Our results show that the method is effective for 4D shape reconstructions of swimming fish, with greater fidelity than other state-of-the-art algorithms.</dcterms:abstract>
    <dcterms:issued>2022</dcterms:issued>
    <dcterms:title>DeepShapeKit : accurate 4D shape reconstruction of swimming fish</dcterms:title>
  </rdf:Description>
</rdf:RDF>
Internal note
xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter
Contact
URL of original publication
Test date of URL
Examination date of dissertation
Method of financing
Comment on publication
Alliance license
Corresponding Authors der Uni Konstanz vorhanden
International Co-Authors
Bibliography of Konstanz
Yes
Refereed