TOWARD SYMBIOTIC HUMAN-AI INTERACTION FOCUSING ON PROGRAMMING BY EXAMPLE

Abstract

Programming has become a new literacy, but is still inaccessible to ordinary people. Programming-by-example (PBE) is an alternative approach that allows people to teach computers repetitive tasks by demonstrating couple input and output examples of the tasks. While the advancements of PBE have been mainly driven by algorithmic improvements, a growing community of researchers started realizing the importance of issues on the human side of PBE. For instance, inexperienced users often find it hard to provide complete and consistent examples, which is crucial for computers to learn the correct programs. Unfortunately, most PBE systems have limited ways to communicate with users about what it can or cannot do, and how to handle unsuccessful situations. The lack of symbiotic interaction between human users and PBE engines remain as a major hurdle against a widespread adoption of PBE techniques.

To address the issues on the human side of PBE, this dissertation has four research threads. First, we began with two formative studies to establish a better understanding of inexperienced users' needs and mental models. Second, based on the findings of the formative studies, we developed a Visual Environment for Symbiotic Programming, called VESPY. VESPY interleaves visual programming and PBE techniques, enabling users (1) to decompose complex tasks into small modules on its 2-d grid, and (2) to complete each module by providing input and output examples. Four sample programs demonstrate VESPY's remarkable versatility. However, we also noticed that VESPY still had a number of usability issues. Third, to better understand the usability issues and how to help users out from common mistakes, we conducted an online user study that observed how inexperience users perform program decomposition and disambiguation, which are the two core activities of PBE. We identified seven types of mistakes, and reaffirmed that informative feedback on those mistakes is crucial for designing usable systems. Finally, we explored the design space of feedback components, in order to understand their impact on user's experience.

My dissertation contributes to the AI and HCI communities with: (i) identification of unmet needs of end-users of the Web; (ii) characterization of non-programmers’ mental model; (iii) design process of interleaving visual programming and PBE; (iv) identification of mistakes people make while using PBE; and (v) design and assessment of feedback components for PBE users.