Change Detection: Theoretical and Applied Approaches for Providing Updates Related to a Topic of Interest

Abstract

The type of user studied in this dissertation has built up expertise on a topic of interest to them, and regularly invests time to find updates on that topic. This research area—referred to within this dissertation as "change detection"—includes the user's process of identifying what has changed as well as internalizing the changes into their mental model. For these users who follow a specific topic over time, how might a system organize information to enable them to update their mental model quickly? Current information retrieval systems are largely not optimized for addressing the long-term change detection needs of users. This dissertation focuses on approaches for enhancing the change detection process, including for short documents (e.g., social media) as well as longer documents (e.g., news articles). This mixed methods exploration of change detection consists of four sections. First, this dissertation introduces a new theory: the Group-Pile-Arrange (GPA) Change Detection Theory. This theory is about organizing documents relevant to a topic of interest in order to accelerate an individual's ability to identify changes and update their mental model. The three components of this theory include: 1. Group the documents by theme; 2. Pile the grouped documents into an order; and 3. Arrange the piles in a meaningful way for the user. These steps could be applied in a range of ways, including using approaches driven by people (e.g., a research librarian providing information), computers (e.g., an information retrieval system), or a hybrid of the two. The second section of this dissertation includes the results of a survey on users' sort order preferences in social media. For this study, change detection was compared with three other use cases: following an event while it happens (experiential), running a search within social media, and browsing social media posts. Respondents recognized the change detection use case, with 66% of the respondents indicating that they perform change detection tasks on social media sites. When engaged in change detection tasks, these respondents showed a strong preference for posts to be clustered and presented in reverse chronological order, in alignment with the "group" and "pile" components of the GPA Change Detection Theory. These organization preferences were distinct from the other studied use cases.

To further understand users' goals and preferences related to change detection, the third section of this dissertation includes the design and prototype implementation of a change detection system called Daybreak. The Daybreak system presents news articles relevant to a user's topic of interest and allows them to tag articles and apply tag labels. Based on these tags and tag labels, the system retrieves new results, groups them into subtopic clusters based on the user's tags, enables generation of chronological or relevance-based piles of documents, and arranges the piles by subtopic importance; for this study, rarity was used as a proxy for subtopic importance. The Daybreak system was used for a qualitative user study, using the framework method for analyzing and interpreting results. In this study, fifteen participants engaged in a change detection scenario across five simulated "days." The participants heavily leveraged the Daybreak system's clustering function when viewing results; there was a weak preference for chronological sorting of documents, compared to relevance ranking. The participants did not view rarity as an effective proxy for subtopic importance; instead, they preferred approaches that enabled them to indicate which subtopics were of greatest interest, such as pinning certain subtopics.

The fourth and final component of this dissertation research describes an evaluation approach for comparing arrangements of subtopic clusters (piles). This evaluation approach uses Spearman's rank correlation coefficient to compare a user's ideal subtopic ordering with a variety of system-generated orderings. This includes a sample evaluation using data from the Daybreak user study to demonstrate how a formal evaluation would work.

Based on the results of these four dissertation research components, it appears that the GPA Change Detection Theory provides a useful framework for organizing information for individuals engaged in change detection tasks. This research provides insights into users' change detection needs and behaviors that could be helpful for building or extending systems attempting to address this use case.