The role of multimodal cues in second language comprehension

Lade...
Vorschaubild
Dateien
Zhang_2-vdvuhuc65spz8.pdf
Zhang_2-vdvuhuc65spz8.pdfGröße: 1.25 MBDownloads: 8
Datum
2023
Autor:innen
Zhang, Ye
Ding, Rong
Tuomainen, Jyrki
Klavinskis-Whiting, Sebastian
Vigliocco, Gabriella
Herausgeber:innen
Kontakt
ISSN der Zeitschrift
Electronic ISSN
ISBN
Bibliografische Daten
Verlag
Schriftenreihe
Auflagebezeichnung
ArXiv-ID
Internationale Patentnummer
Link zur Lizenz
Angaben zur Forschungsförderung
European Union (EU): 743035
European Union (EU): 743035
Projekt
Open Access-Veröffentlichung
Open Access Gold
Sammlungen
Core Facility der Universität Konstanz
Gesperrt bis
Titel in einer weiteren Sprache
Forschungsvorhaben
Organisationseinheiten
Zeitschriftenheft
Publikationstyp
Zeitschriftenartikel
Publikationsstatus
Published
Erschienen in
Scientific Reports. Springer. 2023, 13, 20824. eISSN 2045-2322. Available under: doi: 10.1038/s41598-023-47643-2
Zusammenfassung

In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues – meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal – while using multimodal cues to a lesser extent than L1 comprehenders overall.

Zusammenfassung in einer weiteren Sprache
Fachgebiet (DDC)
400 Sprachwissenschaft, Linguistik
Schlagwörter
Konferenz
Rezension
undefined / . - undefined, undefined
Zitieren
ISO 690ZHANG, Ye, Rong DING, Diego FRASSINELLI, Jyrki TUOMAINEN, Sebastian KLAVINSKIS-WHITING, Gabriella VIGLIOCCO, 2023. The role of multimodal cues in second language comprehension. In: Scientific Reports. Springer. 2023, 13, 20824. eISSN 2045-2322. Available under: doi: 10.1038/s41598-023-47643-2
BibTex
@article{Zhang2023multi-69026,
  year={2023},
  doi={10.1038/s41598-023-47643-2},
  title={The role of multimodal cues in second language comprehension},
  volume={13},
  journal={Scientific Reports},
  author={Zhang, Ye and Ding, Rong and Frassinelli, Diego and Tuomainen, Jyrki and Klavinskis-Whiting, Sebastian and Vigliocco, Gabriella},
  note={Article Number: 20824}
}
RDF
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/69026">
    <dc:rights>Attribution 4.0 International</dc:rights>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2024-01-12T08:58:40Z</dc:date>
    <dc:contributor>Ding, Rong</dc:contributor>
    <dcterms:rights rdf:resource="http://creativecommons.org/licenses/by/4.0/"/>
    <dc:creator>Zhang, Ye</dc:creator>
    <dc:contributor>Tuomainen, Jyrki</dc:contributor>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2024-01-12T08:58:40Z</dcterms:available>
    <dc:creator>Tuomainen, Jyrki</dc:creator>
    <dc:contributor>Vigliocco, Gabriella</dc:contributor>
    <dcterms:abstract>In face-to-face communication, multimodal cues such as prosody, gestures, and mouth movements can play a crucial role in language processing. While several studies have addressed how these cues contribute to native (L1) language processing, their impact on non-native (L2) comprehension is largely unknown. Comprehension of naturalistic language by L2 comprehenders may be supported by the presence of (at least some) multimodal cues, as these provide correlated and convergent information that may aid linguistic processing. However, it is also the case that multimodal cues may be less used by L2 comprehenders because linguistic processing is more demanding than for L1 comprehenders, leaving more limited resources for the processing of multimodal cues. In this study, we investigated how L2 comprehenders use multimodal cues in naturalistic stimuli (while participants watched videos of a speaker), as measured by electrophysiological responses (N400) to words, and whether there are differences between L1 and L2 comprehenders. We found that prosody, gestures, and informative mouth movements each reduced the N400 in L2, indexing easier comprehension. Nevertheless, L2 participants showed weaker effects for each cue compared to L1 comprehenders, with the exception of meaningful gestures and informative mouth movements. These results show that L2 comprehenders focus on specific multimodal cues – meaningful gestures that support meaningful interpretation and mouth movements that enhance the acoustic signal – while using multimodal cues to a lesser extent than L1 comprehenders overall.</dcterms:abstract>
    <dc:contributor>Klavinskis-Whiting, Sebastian</dc:contributor>
    <dc:creator>Ding, Rong</dc:creator>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/69026"/>
    <dc:creator>Klavinskis-Whiting, Sebastian</dc:creator>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dc:language>eng</dc:language>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
    <dc:creator>Vigliocco, Gabriella</dc:creator>
    <dc:contributor>Frassinelli, Diego</dc:contributor>
    <dc:contributor>Zhang, Ye</dc:contributor>
    <dcterms:title>The role of multimodal cues in second language comprehension</dcterms:title>
    <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/69026/1/Zhang_2-vdvuhuc65spz8.pdf"/>
    <dcterms:issued>2023</dcterms:issued>
    <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/69026/1/Zhang_2-vdvuhuc65spz8.pdf"/>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dc:creator>Frassinelli, Diego</dc:creator>
  </rdf:Description>
</rdf:RDF>
Interner Vermerk
xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter
Kontakt
URL der Originalveröffentl.
Prüfdatum der URL
Prüfungsdatum der Dissertation
Finanzierungsart
Kommentar zur Publikation
Allianzlizenz
Corresponding Authors der Uni Konstanz vorhanden
Internationale Co-Autor:innen
Universitätsbibliographie
Ja
Begutachtet
Ja
Diese Publikation teilen