Publikation:

LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores

Lade...
Vorschaubild

Dateien

Sevastjanova_2-ffvtjpdbronh6.pdf
Sevastjanova_2-ffvtjpdbronh6.pdfGröße: 11.15 MBDownloads: 22

Datum

2022

Herausgeber:innen

Kontakt

ISSN der Zeitschrift

Electronic ISSN

ISBN

Bibliografische Daten

Verlag

Schriftenreihe

Auflagebezeichnung

DOI (zitierfähiger Link)
ArXiv-ID

Internationale Patentnummer

Link zur Lizenz

Angaben zur Forschungsförderung

Projekt

Open Access-Veröffentlichung
Open Access Hybrid
Core Facility der Universität Konstanz

Gesperrt bis

Titel in einer weiteren Sprache

Publikationstyp
Zeitschriftenartikel
Publikationsstatus
Published

Erschienen in

Computer Graphics Forum. Wiley. 2022, 41(3), pp. 295-307. ISSN 0167-7055. eISSN 1467-8659. Available under: doi: 10.1111/cgf.14541

Zusammenfassung

Language models, such as BERT, construct multiple, contextualized embeddings for each word occurrence in a corpus. Understanding how the contextualization propagates through the model's layers is crucial for deciding which layers to use for a specific analysis task. Currently, most embedding spaces are explained by probing classifiers; however, some findings remain inconclusive. In this paper, we present LMFingerprints, a novel scoring-based technique for the explanation of contextualized word embeddings. We introduce two categories of scoring functions, which measure (1) the degree of contextualization, i.e., the layerwise changes in the embedding vectors, and (2) the type of contextualization, i.e., the captured context information. We integrate these scores into an interactive explanation workspace. By combining visual and verbal elements, we provide an overview of contextualization in six popular transformer-based language models. We evaluate hypotheses from the domain of computational linguistics, and our results not only confirm findings from related work but also reveal new aspects about the information captured in the embedding spaces. For instance, we show that while numbers are poorly contextualized, stopwords have an unexpected high contextualization in the models' upper layers, where their neighborhoods shift from similar functionality tokens to tokens that contribute to the meaning of the surrounding sentences.

Zusammenfassung in einer weiteren Sprache

Fachgebiet (DDC)
400 Sprachwissenschaft, Linguistik

Schlagwörter

Konferenz

Rezension
undefined / . - undefined, undefined

Forschungsvorhaben

Organisationseinheiten

Zeitschriftenheft

Zugehörige Datensätze in KOPS

Zitieren

ISO 690SEVASTJANOVA, Rita, Aikaterini-Lida KALOULI, Christin SCHÄTZLE, Hanna SCHÄFER, Mennatallah EL-ASSADY, 2022. LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores. In: Computer Graphics Forum. Wiley. 2022, 41(3), pp. 295-307. ISSN 0167-7055. eISSN 1467-8659. Available under: doi: 10.1111/cgf.14541
BibTex
@article{Sevastjanova2022-07-29LMFin-58281,
  year={2022},
  doi={10.1111/cgf.14541},
  title={LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores},
  number={3},
  volume={41},
  issn={0167-7055},
  journal={Computer Graphics Forum},
  pages={295--307},
  author={Sevastjanova, Rita and Kalouli, Aikaterini-Lida and Schätzle, Christin and Schäfer, Hanna and El-Assady, Mennatallah},
  note={Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) within projects BU 1806/10-2 “Questions Visualized” of the FOR2111 and project D02 “Evaluation Metrics for Visual Analytics in Linguistics” (Project ID: 251654672 – TRR 161)}
}
RDF
<rdf:RDF
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:bibo="http://purl.org/ontology/bibo/"
    xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#"
    xmlns:foaf="http://xmlns.com/foaf/0.1/"
    xmlns:void="http://rdfs.org/ns/void#"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > 
  <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/58281">
    <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/58281/1/Sevastjanova_2-ffvtjpdbronh6.pdf"/>
    <dc:creator>Kalouli, Aikaterini-Lida</dc:creator>
    <dcterms:rights rdf:resource="http://creativecommons.org/licenses/by-nc-nd/4.0/"/>
    <dcterms:issued>2022-07-29</dcterms:issued>
    <dc:contributor>El-Assady, Mennatallah</dc:contributor>
    <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-08-12T09:23:52Z</dcterms:available>
    <dc:contributor>Schätzle, Christin</dc:contributor>
    <dc:creator>Schäfer, Hanna</dc:creator>
    <dc:contributor>Kalouli, Aikaterini-Lida</dc:contributor>
    <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/>
    <dcterms:title>LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores</dcterms:title>
    <foaf:homepage rdf:resource="http://localhost:8080/"/>
    <dc:contributor>Schäfer, Hanna</dc:contributor>
    <dc:language>eng</dc:language>
    <dc:creator>Sevastjanova, Rita</dc:creator>
    <dcterms:abstract xml:lang="eng">Language models, such as BERT, construct multiple, contextualized embeddings for each word occurrence in a corpus. Understanding how the contextualization propagates through the model's layers is crucial for deciding which layers to use for a specific analysis task. Currently, most embedding spaces are explained by probing classifiers; however, some findings remain inconclusive. In this paper, we present LMFingerprints, a novel scoring-based technique for the explanation of contextualized word embeddings. We introduce two categories of scoring functions, which measure (1) the degree of contextualization, i.e., the layerwise changes in the embedding vectors, and (2) the type of contextualization, i.e., the captured context information. We integrate these scores into an interactive explanation workspace. By combining visual and verbal elements, we provide an overview of contextualization in six popular transformer-based language models. We evaluate hypotheses from the domain of computational linguistics, and our results not only confirm findings from related work but also reveal new aspects about the information captured in the embedding spaces. For instance, we show that while numbers are poorly contextualized, stopwords have an unexpected high contextualization in the models' upper layers, where their neighborhoods shift from similar functionality tokens to tokens that contribute to the meaning of the surrounding sentences.</dcterms:abstract>
    <dc:creator>El-Assady, Mennatallah</dc:creator>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/>
    <dc:contributor>Sevastjanova, Rita</dc:contributor>
    <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/58281"/>
    <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
    <dc:creator>Schätzle, Christin</dc:creator>
    <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/58281/1/Sevastjanova_2-ffvtjpdbronh6.pdf"/>
    <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-08-12T09:23:52Z</dc:date>
    <dc:rights>Attribution-NonCommercial-NoDerivatives 4.0 International</dc:rights>
    <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/45"/>
  </rdf:Description>
</rdf:RDF>

Interner Vermerk

xmlui.Submission.submit.DescribeStep.inputForms.label.kops_note_fromSubmitter

Kontakt
URL der Originalveröffentl.

Prüfdatum der URL

Prüfungsdatum der Dissertation

Finanzierungsart

Kommentar zur Publikation

Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) within projects BU 1806/10-2 “Questions Visualized” of the FOR2111 and project D02 “Evaluation Metrics for Visual Analytics in Linguistics” (Project ID: 251654672 – TRR 161)
Allianzlizenz
Corresponding Authors der Uni Konstanz vorhanden
Internationale Co-Autor:innen
Universitätsbibliographie
Ja
Begutachtet
Ja
Diese Publikation teilen