PROCESS CONFORMANCE TESTING: A METHODOLOGY TO IDENTIFY AND UNDERSTAND PROCESS VIOLATIONS IN ENACTMENT OF SOFTWARE PROCESSES

Abstract

Today's software development is driven by software processes and practices that when followed increase the chances of building high quality software products. Not following these guidelines results in increased risk that the goal for the software's quality characteristics cannot be reached. Current process analysis approaches are limited in identifying and understanding process deviations and ultimately fail in comprehending why a process does not work in a given environment and what steps of the process have to be changed and tailored. In this work I will present a methodology for formulating, identifying and investigating process violations in the execution of software processes. The methodology, which can be thought of as "Process Conformance Testing", consists of a four step iterative model, compromising templates and tools. A strong focus is set on identifying violations in a cost efficient and unobtrusive manner by utilizing automatically collected data gathered through commonly used software development tools, such as version control systems. To evaluate the usefulness and correctness of the model a series of four studies have been conducted in both classroom and professional environments. A total of eight different software processes have been investigated and tested. The results of the studies show that the steps and iterative character of the methodology are useful for formulating and tailoring violation detection strategies and investigating violations in classroom study environments and professional environments. All the investigated processes were violated in some way, which emphasizes the importance of conformance measurement. This is especially important when running an empirical study to evaluate the effectiveness of a software process, as the experimenters want to make sure they are evaluating the specified process and not a variation of it. Violation detection strategies were tailored based upon analysis of the history of violations and feedback from then enactors and mangers yielding greater precision of identification of non-conformities. The overhead cost of the approach is shown to be feasible with a 3.4% (professional environment) and 12.1% (classroom environment) overhead. One interesting side result is that process enactors did not always follow the process for good reason, e.g. the process was not tailored for the environment, it was not specified at the right level of granularity, or was too difficult to follow. Two specific examples in this thesis are XP Pair Switching and Test Driven Development. In XP Pair Switching, the practice was violated because the frequency of switching was too high. The definition of Test Driven Development is simple and clear but requires a fair amount of discipline to follow, especially by novice programmers.