Accounting for Defect Characteristics in Empirical Studies of Software Testing

Abstract

Software testing is an indispensable activity in quality assurance and an enduring topic of research. For decades, researchers have been inventing new and better techniques to test software. However, no testing technique will ever be a panacea for all software defects. Therefore, while researchers should continue to develop new testing techniques, they also need to deeply understand the abilities and limitations of existing techniques, the ways they complement each other, and the trade-offs involved in using different techniques. This work contends that researchers cannot sufficiently understand software testing without also understanding software defects. This work is the first to show that simple, automatically-measurable characteristics of defects affect their susceptibility to detection by software testing. Unlike previous attempts to characterize defects, this work offers a characterization that is objective, practical, and proven to help explain why some defects and not others are detected by testing. More importantly, this work shows that researchers can and should account for defect characteristics when they study the effectiveness of software-testing techniques. An experiment methodology is presented that enables experimenters to compare the effectiveness of different techniques and, at the same time, to measure the influence of defect characteristics and other factors on the results. The methodology is demonstrated in a large experiment in the domain of graphical-user-interface testing. As the experiment shows, researchers who use the methodology will understand what kinds of defects tend to be detected by testing and what testing techniques are better at detecting certain kinds of defects. This information can help researchers develop more effective testing techniques, and it can help software testers make better choices about the testing techniques to use on their projects. As this work explains, it also has the potential to help testers detect more defects, and more important defects, during regression testing.