Publikation: More than words : word predictability, prosody, gesture and mouth movements in natural language comprehension
Dateien
Datum
Autor:innen
Herausgeber:innen
ISSN der Zeitschrift
Electronic ISSN
ISBN
Bibliografische Daten
Verlag
Schriftenreihe
Auflagebezeichnung
URI (zitierfähiger Link)
DOI (zitierfähiger Link)
Internationale Patentnummer
Link zur Lizenz
Angaben zur Forschungsförderung
Projekt
Open Access-Veröffentlichung
Sammlungen
Core Facility der Universität Konstanz
Titel in einer weiteren Sprache
Publikationstyp
Publikationsstatus
Erschienen in
Zusammenfassung
The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.
Zusammenfassung in einer weiteren Sprache
Fachgebiet (DDC)
Schlagwörter
Konferenz
Rezension
Zitieren
ISO 690
ZHANG, Ye, Diego FRASSINELLI, Jyrki TUOMAINEN, Jeremy I. SKIPPER, Gabriella VIGLIOCCO, 2021. More than words : word predictability, prosody, gesture and mouth movements in natural language comprehension. In: Proceedings of the Royal Society of London, Series B : Biological Sciences. Royal Society of London. 2021, 288(1955), 20210500. ISSN 0962-8452. eISSN 1471-2954. Available under: doi: 10.1098/rspb.2021.0500BibTex
@article{Zhang2021words-54568, year={2021}, doi={10.1098/rspb.2021.0500}, title={More than words : word predictability, prosody, gesture and mouth movements in natural language comprehension}, number={1955}, volume={288}, issn={0962-8452}, journal={Proceedings of the Royal Society of London, Series B : Biological Sciences}, author={Zhang, Ye and Frassinelli, Diego and Tuomainen, Jyrki and Skipper, Jeremy I. and Vigliocco, Gabriella}, note={Article Number: 20210500} }
RDF
<rdf:RDF xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:bibo="http://purl.org/ontology/bibo/" xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:void="http://rdfs.org/ns/void#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/54568"> <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/54568"/> <dc:language>eng</dc:language> <dc:contributor>Frassinelli, Diego</dc:contributor> <foaf:homepage rdf:resource="http://localhost:8080/"/> <dc:creator>Tuomainen, Jyrki</dc:creator> <dc:rights>terms-of-use</dc:rights> <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/> <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/54568/1/Zhang_2-1pekl5l7fcd8p3.pdf"/> <dc:contributor>Vigliocco, Gabriella</dc:contributor> <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/> <dc:contributor>Skipper, Jeremy I.</dc:contributor> <dcterms:issued>2021</dcterms:issued> <dcterms:abstract xml:lang="eng">The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.</dcterms:abstract> <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/> <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/> <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/54568/1/Zhang_2-1pekl5l7fcd8p3.pdf"/> <dc:creator>Vigliocco, Gabriella</dc:creator> <dc:contributor>Tuomainen, Jyrki</dc:contributor> <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-08-12T07:05:14Z</dcterms:available> <dc:creator>Frassinelli, Diego</dc:creator> <dc:creator>Skipper, Jeremy I.</dc:creator> <dc:contributor>Zhang, Ye</dc:contributor> <dcterms:title>More than words : word predictability, prosody, gesture and mouth movements in natural language comprehension</dcterms:title> <dc:creator>Zhang, Ye</dc:creator> <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-08-12T07:05:14Z</dc:date> </rdf:Description> </rdf:RDF>