Unlocking Tool-agnostic Visualization Applications through Fine-grained Computational Representations

Abstract

Data visualizations are vital tools for data analysis, sensemaking, and storytelling. While many approaches and systems have been developed to enhance visualization understanding and interaction - including classification, authoring, editing, and animation - their effectiveness is often limited by tool-specific constraints. Most existing solutions are tightly coupled with particular visualization tools or formats, making their generalization to visualizations created by other tools challenging and impractical.

This dissertation first presents Mystique, a mixed-initiative authoring tool for decomposing and reusing tool-agnostic SVG charts. It represents our initial investigation into building a specific tool-agnostic visualization application. The development of Mystique helps reveal two major challenges: (1) the lack of a unified and appropriate computational representation for data visualizations, and (2) the lack of visualization corpora containing diverse charts and sufficient semantic labels. To tackle the first challenge, this dissertation contributes Manipulable Semantic Components (MSC), a computational representation framework for visualization scenes. MSC includes both a visualization object model for describing multi-level visualization scenes using fine-grained semantic components, and a set of operations for convenient scene manipulation. Its usability is demonstrated through four visualization applications. To tackle the second challenge, this dissertation first contributes a state-of-the-art survey on visualization corpora in automated chart analysis research to understand common practices of creating chart corpora and the desired properties of benchmark corpora. Based on this knowledge, this dissertation further introduces a diverse chart corpus, VisAnatomy, with fine-grained MSC-informed semantic labels that support automated visualization understanding through modern computational models including graph neural networks and large language models. These contributions demonstrate how a unified computational representation and a chart corpus with fine-grained semantic labels can enable tool-agnostic visualization applications and advance the field of automated chart understanding in the era of AI.