Publikation:

Explaining Contextualization in Language Models using Visual Analytics

Lade...
Vorschaubild

Dateien

Zu diesem Dokument gibt es keine Dateien.

Datum

2021

Herausgeber:innen

Kontakt

ISSN der Zeitschrift

Electronic ISSN

ISBN

Bibliografische Daten

Verlag

Schriftenreihe

Auflagebezeichnung

URI (zitierfähiger Link)
ArXiv-ID

Internationale Patentnummer

Angaben zur Forschungsförderung

Projekt

Open Access-Veröffentlichung
Core Facility der Universität Konstanz

Gesperrt bis

Titel in einer weiteren Sprache

Publikationstyp
Beitrag zu einem Konferenzband
Publikationsstatus
Published

Erschienen in

ZONG, Chengqing, ed., Fei XIA, ed., Wenjie LI, ed. and others. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Volume 1: Long Papers). Stroudsburg, PA: Association for Computational Linguistics, 2021, pp. 464-476. ISBN 978-1-954085-52-7. Available under: doi: 10.18653/v1/2021.acl-long.39

Zusammenfassung

Despite the success of contextualized language models on various NLP tasks, it is still unclear what these models really learn. In this paper, we contribute to the current efforts of explaining such models by exploring the continuum between function and content words with respect to contextualization in BERT, based on linguistically-informed insights. In particular, we utilize scoring and visual analytics techniques: we use an existing similarity-based score to measure contextualization and integrate it into a novel visual analytics technique, presenting the model’s layers simultaneously and highlighting intra-layer properties and inter-layer differences. We show that contextualization is neither driven by polysemy nor by pure context variation. We also provide insights on why BERT fails to model words in the middle of the functionality continuum.

Zusammenfassung in einer weiteren Sprache

Fachgebiet (DDC)
400 Sprachwissenschaft, Linguistik

Schlagwörter

Konferenz

59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 1. Aug. 2021 - 6. Aug. 2021, Bangkok
Rezension
undefined / . - undefined, undefined

Forschungsvorhaben

Organisationseinheiten

Zeitschriftenheft

Zugehörige Datensätze in KOPS

Zitieren

ISO 690SEVASTJANOVA, Rita, Aikaterini-Lida KALOULI, Christin SCHÄTZLE, Hanna SCHÄFER, Mennatallah EL-ASSADY, 2021. Explaining Contextualization in Language Models using Visual Analytics. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Bangkok, 1. Aug. 2021 - 6. Aug. 2021. In: ZONG, Chengqing, ed., Fei XIA, ed., Wenjie LI, ed. and others. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Volume 1: Long Papers). Stroudsburg, PA: Association for Computational Linguistics, 2021, pp. 464-476. ISBN 978-1-954085-52-7. Available under: doi: 10.18653/v1/2021.acl-long.39
BibTex
@inproceedings{Sevastjanova2021Expla-55999,
  year={2021},
  doi={10.18653/v1/2021.acl-long.39},
  title={Explaining Contextualization in Language Models using Visual Analytics},
  isbn={978-1-954085-52-7},
  publisher={Association for Computational Linguistics},
  address={Stroudsburg, PA},
  booktitle={Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Volume 1: Long Papers)},
  pages={464--476},
  editor={Zong, Chengqing and Xia, Fei and Li, Wenjie},
  author={Sevastjanova, Rita and Kalouli, Aikaterini-Lida and Schätzle, Christin and Schäfer, Hanna and El-Assady, Mennatallah}
}
RDF
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/55999">
    <dcterms:rights rdf:resource="https://rightsstatements.org/page/InC/1.0/"/>
    <dc:contributor>Kalouli, Aikaterini-Lida</dc:contributor>
    <dc:contributor>Schätzle, Christin</dc:contributor>
    <dc:creator>Schätzle, Christin</dc:creator>
    <dc:creator>Sevastjanova, Rita</dc:creator>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dc:creator>Kalouli, Aikaterini-Lida</dc:creator>
    <dc:creator>Schäfer, Hanna</dc:creator>
    <dc:contributor>Schäfer, Hanna</dc:contributor>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-12-22T13:17:02Z</dcterms:available>
    <dc:contributor>El-Assady, Mennatallah</dc:contributor>
    <dcterms:issued>2021</dcterms:issued>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/55999"/>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2021-12-22T13:17:02Z</dc:date>
    <dc:creator>El-Assady, Mennatallah</dc:creator>
    <dc:rights>terms-of-use</dc:rights>
    <dcterms:abstract xml:lang="eng">Despite the success of contextualized language models on various NLP tasks, it is still unclear what these models really learn. In this paper, we contribute to the current efforts of explaining such models by exploring the continuum between function and content words with respect to contextualization in BERT, based on linguistically-informed insights. In particular, we utilize scoring and visual analytics techniques: we use an existing similarity-based score to measure contextualization and integrate it into a novel visual analytics technique, presenting the model’s layers simultaneously and highlighting intra-layer properties and inter-layer differences. We show that contextualization is neither driven by polysemy nor by pure context variation. We also provide insights on why BERT fails to model words in the middle of the functionality continuum.</dcterms:abstract>
    <dc:contributor>Sevastjanova, Rita</dc:contributor>
    <dcterms:title>Explaining Contextualization in Language Models using Visual Analytics</dcterms:title>
    <dc:language>eng</dc:language>
  </rdf:Description>
</rdf:RDF>

Interner Vermerk

xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter

Kontakt
URL der Originalveröffentl.

Prüfdatum der URL

Prüfungsdatum der Dissertation

Finanzierungsart

Kommentar zur Publikation

Allianzlizenz
Corresponding Authors der Uni Konstanz vorhanden
Internationale Co-Autor:innen
Universitätsbibliographie
Ja
Begutachtet
Diese Publikation teilen