Publikation: VISITOR : Visual Interactive State Sequence Exploration for Reinforcement Learning
Dateien
Datum
Autor:innen
Herausgeber:innen
ISSN der Zeitschrift
Electronic ISSN
ISBN
Bibliografische Daten
Verlag
Schriftenreihe
Auflagebezeichnung
URI (zitierfähiger Link)
DOI (zitierfähiger Link)
Internationale Patentnummer
Link zur Lizenz
Angaben zur Forschungsförderung
Projekt
Open Access-Veröffentlichung
Core Facility der Universität Konstanz
Titel in einer weiteren Sprache
Publikationstyp
Publikationsstatus
Erschienen in
Zusammenfassung
Understanding the behavior of deep reinforcement learning agents is a crucial requirement throughout their development. Existing work has addressed the identification of observable behavioral patterns in state sequences or analysis of isolated internal representations; however, the overall decision-making of deep-learning RL agents remains opaque. To tackle this, we present VISITOR, a visual analytics system enabling the analysis of entire state sequences, the diagnosis of singular predictions, and the comparison between agents. A sequence embedding view enables the multiscale analysis of state sequences, utilizing custom embedding techniques for a stable spatialization of the observations and internal states. We provide multiple layers: (1) a state space embedding, highlighting different groups of states inside the state-action sequences, (2) a trajectory view, emphasizing decision points, (3) a network activation mapping, visualizing the relationship between observations and network activations, (4) a transition embedding, enabling the analysis of state-to-state transitions. The embedding view is accompanied by an interactive reward view that captures the temporal development of metrics, which can be linked directly to states in the embedding. Lastly, a model list allows for the quick comparison of models across multiple metrics. Annotations can be exported to communicate results to different audiences. Our two-stage evaluation with eight experts confirms the effectiveness in identifying states of interest, comparing the quality of policies, and reasoning about the internal decision-making processes.
Zusammenfassung in einer weiteren Sprache
Fachgebiet (DDC)
Schlagwörter
Konferenz
Rezension
Zitieren
ISO 690
METZ, Yannick, Eugene BYKOVETS, Lucas JOOS, Daniel A. KEIM, Mennatallah EL-ASSADY, 2023. VISITOR : Visual Interactive State Sequence Exploration for Reinforcement Learning. In: Computer Graphics Forum. Wiley. 2023, 42(3), S. 397-408. ISSN 0167-7055. eISSN 1467-8659. Verfügbar unter: doi: 10.1111/cgf.14839BibTex
@article{Metz2023-06VISIT-67521, year={2023}, doi={10.1111/cgf.14839}, title={VISITOR : Visual Interactive State Sequence Exploration for Reinforcement Learning}, number={3}, volume={42}, issn={0167-7055}, journal={Computer Graphics Forum}, pages={397--408}, author={Metz, Yannick and Bykovets, Eugene and Joos, Lucas and Keim, Daniel A. and El-Assady, Mennatallah} }
RDF
<rdf:RDF xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:bibo="http://purl.org/ontology/bibo/" xmlns:dspace="http://digital-repositories.org/ontologies/dspace/0.1.0#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:void="http://rdfs.org/ns/void#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" > <rdf:Description rdf:about="https://kops.uni-konstanz.de/server/rdf/resource/123456789/67521"> <dc:creator>Joos, Lucas</dc:creator> <dc:contributor>El-Assady, Mennatallah</dc:contributor> <dcterms:issued>2023-06</dcterms:issued> <dc:contributor>Metz, Yannick</dc:contributor> <dspace:isPartOfCollection rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/> <dcterms:title>VISITOR : Visual Interactive State Sequence Exploration for Reinforcement Learning</dcterms:title> <dc:contributor>Keim, Daniel A.</dc:contributor> <bibo:uri rdf:resource="https://kops.uni-konstanz.de/handle/123456789/67521"/> <dcterms:available rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-08-07T14:53:20Z</dcterms:available> <dc:creator>Keim, Daniel A.</dc:creator> <dcterms:hasPart rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/67521/1/Metz_2-1b7letggi23f48.pdf"/> <dcterms:abstract>Understanding the behavior of deep reinforcement learning agents is a crucial requirement throughout their development. Existing work has addressed the identification of observable behavioral patterns in state sequences or analysis of isolated internal representations; however, the overall decision-making of deep-learning RL agents remains opaque. To tackle this, we present VISITOR, a visual analytics system enabling the analysis of entire state sequences, the diagnosis of singular predictions, and the comparison between agents. A sequence embedding view enables the multiscale analysis of state sequences, utilizing custom embedding techniques for a stable spatialization of the observations and internal states. We provide multiple layers: (1) a state space embedding, highlighting different groups of states inside the state-action sequences, (2) a trajectory view, emphasizing decision points, (3) a network activation mapping, visualizing the relationship between observations and network activations, (4) a transition embedding, enabling the analysis of state-to-state transitions. The embedding view is accompanied by an interactive reward view that captures the temporal development of metrics, which can be linked directly to states in the embedding. Lastly, a model list allows for the quick comparison of models across multiple metrics. Annotations can be exported to communicate results to different audiences. Our two-stage evaluation with eight experts confirms the effectiveness in identifying states of interest, comparing the quality of policies, and reasoning about the internal decision-making processes.</dcterms:abstract> <dc:contributor>Joos, Lucas</dc:contributor> <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2023-08-07T14:53:20Z</dc:date> <dcterms:rights rdf:resource="http://creativecommons.org/licenses/by-nc/4.0/"/> <foaf:homepage rdf:resource="http://localhost:8080/"/> <dspace:hasBitstream rdf:resource="https://kops.uni-konstanz.de/bitstream/123456789/67521/1/Metz_2-1b7letggi23f48.pdf"/> <void:sparqlEndpoint rdf:resource="http://localhost/fuseki/dspace/sparql"/> <dc:creator>Bykovets, Eugene</dc:creator> <dc:rights>Attribution-NonCommercial 4.0 International</dc:rights> <dc:contributor>Bykovets, Eugene</dc:contributor> <dc:creator>El-Assady, Mennatallah</dc:creator> <dc:creator>Metz, Yannick</dc:creator> <dcterms:isPartOf rdf:resource="https://kops.uni-konstanz.de/server/rdf/resource/123456789/36"/> <dc:language>eng</dc:language> </rdf:Description> </rdf:RDF>