Visual Sheet Music Analytics

Abstract

Through integrating visual interactive data analysis with sheet music, this thesis addresses a new interdisciplinary field: Visual Musicology. This work bridges the gap between information visualization and musicology, paving the way for new methods to analyze and interpret music data, specifically focusing on sheet music. The Western common music notation (CMN), used in sheet music, is uncontested among musicologists, educators, students, and musicians. Nevertheless, CMN's intricate and complex design challenges interpreting and analyzing large amounts of sheet music collections, particularly for those who are no visualization experts. We address these challenges by introducing novel visual analytics solutions that extend beyond the scope of existing research in the field. The initial foundational pillar establishes Visual Musicology, where we introduce the Visual Musicology Graph and the Methodology Transfer Model as organizational frameworks stimulating interdisciplinary research collaboration. They facilitate solution transfer across domains, as demonstrated through various musicological scenarios such as embodied music interaction. We also examine existing music notation techniques, discovering opportunities for unapplied designs that replace or connect CMN with advanced visualization techniques. Then, visualization designs for augmenting digital sheet music with harmonic and rhythmic fingerprints based on domain-specific concepts are presented, including the circle of fifths and rhythm tree. We show how to employ melodic operators to trace the progression of melodies in compositions, enabling analysts to understand the melodic relationship of the motivic and thematic material. By augmenting instead of replacing CMN, our research shows that these designs improve the accessibility and understandability of sheet music for people with varying musical and visual analytic expertise. Through active collaboration with musicologists and comprehensive qualitative user studies, our research assesses the applicability and effectiveness of the augmentation strategy. Building on these previous two foundational parts, the thesis proposes two approaches for sheet music analysis: Bottom-Up Composition Analysis (MusicVis) and Top-Down Corpus Analysis (CorpusVis). MusicVis enriches digital sheet music by extending the harmonic, rhythmic, and melodic analysis through flexible visual interactive queries and seamless transitions to abstract analysis levels. Conversely, CorpusVis facilitates macro-level exploration of sheet music collections. Qualitative user studies show that the proposed techniques support the interpretation and understanding of complex musical structures at different levels of abstraction while, for instance, scalability remains a limitation. The thesis concludes by reflecting on lessons learned, such as building on familiar concepts and separating independent data characteristics for analysis. We also address challenges encountered during our conducted Visual Sheet Music Analytics research, like transforming abstract musical concepts into intuitive visual representations while factoring in user requirements. Furthermore, we emphasize specific future research projects in Visual Musicology, such as integrating our proposed fingerprint designs into AR platforms, thus offering a hands-on approach for musicians to interact with augmented sheet music while learning and playing their instruments.