Relevance judgments and query reformulation by users interacting with a speech retrieval system

Abstract

This dissertation presents a framework for searcher behavior that can be used as a basis for designing future speech retrieval systems. It reports on an exploratory study that examines: the criteria searchers of oral history interviews use when judging the relevance of a recording or a passage; the attributes on which those judgments are based; the moves searchers adopt for information need refinements (INR); and the types of query reformulation by which those moves are realized. Eight participants that include faculty, Holocaust scholars, a film producer, and a high school teacher searched the Survivors of the Shoah Visual History Foundation's collection that consists of 116,000 hours of 52,000 testimonies in 32 different languages from the survivors, liberators, rescuers and witnesses of the Holocaust. Each participant performed a series of searches based on his/her own interests over a period of three to nine days. Data were collected through observation and screen capture, think aloud, semi-structured interviews, and focus group discussions; coded; and analyzed by looking for patterns. The cognitive process of relevance judgment and query reformulation occurred interactively during a search. As a result, some relevance criteria (topicality, comprehensibility, novelty of content, and acquaintance) and INR moves (clarification alone, specialization, restriction, and note for later) were observed during both processes. Some criteria, such as richness and emotion, were medium (i.e., speech) and domain (i.e., oral history) specific. The findings identified four different types of attributes of a recording or a passage that included spoken-content attributes (person, place, event/experience, organization/group, ), audio and/or visual attributes (facial expression, voice, gesture, displayed artifact, ), non-content attributes (cache, digitization, language, ), and biographical attributes (name of interviewee, date of birth, gender, occupation, ). Searchers used different query reformulation types, such as adding a condition, narrowing a condition, new term, broadening a condition, removing a condition, and modifying a condition, in order to achieve different INR moves. Some important implications for indexing and metadata assignment, support for search and browsing, and task-oriented system and interface design are drawn from the findings. It then concludes with discussions on limitations and ideas for future work.