Testing GUI-based Software with Undetermined Input Spaces

Abstract

Most software applications feature a Graphical User Interface (GUI) front-end as the main, and often the only, method for the user to interact with the software. System-testing a software application requires it to be tested as a whole through the GUI. Testers need to generate sequences of GUI events (e.g., mouse clicks and menu selections) to exercise various behaviors of the application. Because the input space of a typical GUI (i.e., the space of all possible GUI events and their interactions) is often enormous, manual GUI testing is impractical. Model-based testing is a new approach that automatically and systematically generates a large number of test cases by leveraging a formal model representing the GUI input space. Unfortunately, modern applications often have a ``context-sensitive reachability GUI,'' in which the GUI components are only reachable with some particular state or environment constraints. Thus, it is challenging to determine the GUI input space and and obtain a GUI model for automated GUI testing. This research proposes new testing techniques to tackle the challenges in model-based GUI testing. The central thesis is this: GUI-based applications can be effectively and efficiently tested by systematically and incrementally leveraging the application runtime execution observations. To explore the thesis, a novel model-based testing paradigm called Observer-Model-Exercise* (OME*) is developed. This paradigm relies on the opportunistic observations obtained during test case execution to incrementally explore the GUI input space and construct a GUI model for test case generation. To evaluate OME*, an open-source automated model-based GUI testing framework called GUITAR is developed. An empirical study with 8 widely-used open-source applications demonstrated that the OME* approach is feasible. Compared to previous model-based testing approaches, OME* was able to increase the GUI input space discovered by as much as 1,044%. As a result, 34 new faults were detected in the subject applications.