KOPS - The Institutional Repository of the University of Konstanz

LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores

LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores

Cite This

Files in this item

Files Size Format View

There are no files associated with this item.

SEVASTJANOVA, Rita, Aikaterini-Lida KALOULI, Christin BECK, Hanna HAUPTMANN, Mennatallah EL-ASSADY, 2022. LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores. In: Computer Graphics Forum. Wiley. 41(3), pp. 295-307. ISSN 0167-7055. eISSN 1467-8659. Available under: doi: 10.1111/cgf.14541

@article{Sevastjanova2022-07-29LMFin-58281, title={LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores}, year={2022}, doi={10.1111/cgf.14541}, number={3}, volume={41}, issn={0167-7055}, journal={Computer Graphics Forum}, pages={295--307}, author={Sevastjanova, Rita and Kalouli, Aikaterini-Lida and Beck, Christin and Hauptmann, Hanna and El-Assady, Mennatallah} }

<rdf:RDF xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:bibo="http://purl.org/ontology/bibo/" xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:void="http://rdfs.org/ns/void#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > <rdf:Description rdf:about="https://kops.uni-konstanz.de/rdf/resource/123456789/58281"> <dc:creator>Kalouli, Aikaterini-Lida</dc:creator> <dc:creator>El-Assady, Mennatallah</dc:creator> <dcterms:title>LMFingerprints : Visual Explanations of Language Model Embedding Spaces through Layerwise Contextualization Scores</dcterms:title> <dc:creator>Sevastjanova, Rita</dc:creator> <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/> <dc:contributor>Beck, Christin</dc:contributor> <dc:creator>Beck, Christin</dc:creator> <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-08-12T09:23:52Z</dc:date> <dcterms:abstract xml:lang="eng">Language models, such as BERT, construct multiple, contextualized embeddings for each word occurrence in a corpus. Understanding how the contextualization propagates through the model's layers is crucial for deciding which layers to use for a specific analysis task. Currently, most embedding spaces are explained by probing classifiers; however, some findings remain inconclusive. In this paper, we present LMFingerprints, a novel scoring-based technique for the explanation of contextualized word embeddings. We introduce two categories of scoring functions, which measure (1) the degree of contextualization, i.e., the layerwise changes in the embedding vectors, and (2) the type of contextualization, i.e., the captured context information. We integrate these scores into an interactive explanation workspace. By combining visual and verbal elements, we provide an overview of contextualization in six popular transformer-based language models. We evaluate hypotheses from the domain of computational linguistics, and our results not only confirm findings from related work but also reveal new aspects about the information captured in the embedding spaces. For instance, we show that while numbers are poorly contextualized, stopwords have an unexpected high contextualization in the models' upper layers, where their neighborhoods shift from similar functionality tokens to tokens that contribute to the meaning of the surrounding sentences.</dcterms:abstract> <dcterms:rights rdf:resource="http://creativecommons.org/licenses/by-nc-nd/4.0/"/> <dc:creator>Hauptmann, Hanna</dc:creator> <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/58281"/> <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2022-08-12T09:23:52Z</dcterms:available> <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/rdf/resource/123456789/45"/> <dc:rights>Attribution-NonCommercial-NoDerivatives 4.0 International</dc:rights> <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/rdf/resource/123456789/36"/> <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/rdf/resource/123456789/36"/> <foaf:homepage rdf:resource="http://localhost:8080/jspui"/> <dc:contributor>Sevastjanova, Rita</dc:contributor> <dc:language>eng</dc:language> <dcterms:issued>2022-07-29</dcterms:issued> <dc:contributor>El-Assady, Mennatallah</dc:contributor> <dc:contributor>Kalouli, Aikaterini-Lida</dc:contributor> <dc:contributor>Hauptmann, Hanna</dc:contributor> <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/rdf/resource/123456789/45"/> </rdf:Description> </rdf:RDF>

This item appears in the following Collection(s)

Attribution-NonCommercial-NoDerivatives 4.0 International Except where otherwise noted, this item's license is described as Attribution-NonCommercial-NoDerivatives 4.0 International

Search KOPS


Browse

My Account