Electrophysiological signatures of second language multimodal comprehension

No Thumbnail Available
Files
There are no files associated with this item.
Date
2021
Authors
Zhang, Ye
Ding, Rong
Tuomainen, Jyrki
Klavinskis-Whiting, Sebastian
Vigliocco, Gabriella
Editors
Contact
Journal ISSN
Electronic ISSN
ISBN
Bibliographical data
Publisher
Series
URI (citable link)
DOI (citable link)
ArXiv-ID
International patent number
Link to the license
EU project number
Project
Open Access publication
Collections
Restricted until
Title in another language
Research Projects
Organizational Units
Journal Issue
Publication type
Contribution to a conference collection
Publication status
Published
Published in
Comparative Cognition : Animal Minds : Proceedings of the 43rd Annual Conference of the Cognitive Science Society (CogSci 2021). - Vienna : Cognitive Science Society, 2021. - pp. 2971-2977
Abstract
Language is multimodal: non-linguistic cues, such as prosody, gestures and mouth movements, are always present in face-toface communication and interact to support processing. In this paper, we ask whether and how multimodal cues affect L2 processing by recording EEG for highly proficient bilinguals when watching naturalistic materials. For each word, we quantified surprisal and the informativeness of prosody, gestures, and mouth movements. We found that each cue modulates the N400: prosodic accentuation, meaningful gestures, and informative mouth movements all reduce N400. Further, effects of meaningful gestures but not mouth informativeness are enhanced by prosodic accentuation, whereas effects of mouth are enhanced by meaningful gestures but reduced by beat gestures. Compared with L1, L2 participants benefit less from cues and their interactions, except for meaningful gestures and mouth movements. Thus, in realworld language comprehension, L2 comprehenders use multimodal cues just as L1 speakers albeit to a lesser extent.
Summary in another language
Subject (DDC)
400 Philology, Linguistics
Keywords
multimodal communication; language; N400; gesture; mouth; prosodic accentuation, bilingualism, L2
Conference
43rd Annual Conference of the Cognitive Science Society (CogSci 2021), Jul 26, 2021 - Jul 29, 2021, Vienna, Austria
Review
undefined / . - undefined, undefined. - (undefined; undefined)
Cite This
ISO 690ZHANG, Ye, Rong DING, Diego FRASSINELLI, Jyrki TUOMAINEN, Sebastian KLAVINSKIS-WHITING, Gabriella VIGLIOCCO, 2021. Electrophysiological signatures of second language multimodal comprehension. 43rd Annual Conference of the Cognitive Science Society (CogSci 2021). Vienna, Austria, Jul 26, 2021 - Jul 29, 2021. In: Comparative Cognition : Animal Minds : Proceedings of the 43rd Annual Conference of the Cognitive Science Society (CogSci 2021). Vienna:Cognitive Science Society, pp. 2971-2977
BibTex
@inproceedings{Zhang2021Elect-54388,
  year={2021},
  title={Electrophysiological signatures of second language multimodal comprehension},
  url={https://cognitivesciencesociety.org/wp-content/uploads/2021/07/cogsci21_proceedings_v4.pdf},
  publisher={Cognitive Science Society},
  address={Vienna},
  booktitle={Comparative Cognition : Animal Minds : Proceedings of the 43rd Annual Conference of the Cognitive Science Society (CogSci 2021)},
  pages={2971--2977},
  author={Zhang, Ye and Ding, Rong and Frassinelli, Diego and Tuomainen, Jyrki and Klavinskis-Whiting, Sebastian and Vigliocco, Gabriella}
}
RDF
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/54388">
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/54388"/>
    <dc:contributor>Tuomainen, Jyrki</dc:contributor>
    <dc:creator>Frassinelli, Diego</dc:creator>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
    <dc:contributor>Frassinelli, Diego</dc:contributor>
    <dc:creator>Klavinskis-Whiting, Sebastian</dc:creator>
    <dc:creator>Tuomainen, Jyrki</dc:creator>
    <dc:creator>Ding, Rong</dc:creator>
    <dcterms:abstract xml:lang="eng">Language is multimodal: non-linguistic cues, such as prosody, gestures and mouth movements, are always present in face-toface communication and interact to support processing. In this paper, we ask whether and how multimodal cues affect L2 processing by recording EEG for highly proficient bilinguals when watching naturalistic materials. For each word, we quantified surprisal and the informativeness of prosody, gestures, and mouth movements. We found that each cue modulates the N400: prosodic accentuation, meaningful gestures, and informative mouth movements all reduce N400. Further, effects of meaningful gestures but not mouth informativeness are enhanced by prosodic accentuation, whereas effects of mouth are enhanced by meaningful gestures but reduced by beat gestures. Compared with L1, L2 participants benefit less from cues and their interactions, except for meaningful gestures and mouth movements. Thus, in realworld language comprehension, L2 comprehenders use multimodal cues just as L1 speakers albeit to a lesser extent.</dcterms:abstract>
    <dc:rights>terms-of-use</dc:rights>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dc:language>eng</dc:language>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-07-22T12:39:15Z</dcterms:available>
    <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/>
    <dc:contributor>Zhang, Ye</dc:contributor>
    <dc:contributor>Klavinskis-Whiting, Sebastian</dc:contributor>
    <dc:contributor>Vigliocco, Gabriella</dc:contributor>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-07-22T12:39:15Z</dc:date>
    <dc:creator>Zhang, Ye</dc:creator>
    <dc:contributor>Ding, Rong</dc:contributor>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
    <dc:creator>Vigliocco, Gabriella</dc:creator>
    <dcterms:title>Electrophysiological signatures of second language multimodal comprehension</dcterms:title>
    <dcterms:issued>2021</dcterms:issued>
  </rdf:Description>
</rdf:RDF>
Internal note
xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter
Contact
Examination date of dissertation
Method of financing
Comment on publication
Alliance license
Corresponding Authors der Uni Konstanz vorhanden
International Co-Authors
Bibliography of Konstanz
Yes
Refereed