Software testing is an indispensable activity in quality assurance and

an enduring topic of research. For decades, researchers have been

inventing new and better techniques to test software. However, no

testing technique will ever be a panacea for all software

defects. Therefore, while researchers should continue to develop new

testing techniques, they also need to deeply understand the abilities

and limitations of existing techniques, the ways they complement each

other, and the trade-offs involved in using different techniques. This

work contends that researchers cannot sufficiently understand software

testing without also understanding software defects.

This work is the first to show that simple, automatically-measurable

characteristics of defects affect their susceptibility to detection by

software testing. Unlike previous attempts to characterize defects,

this work offers a characterization that is objective, practical, and

proven to help explain why some defects and not others are detected by

testing.

More importantly, this work shows that researchers can and should

account for defect characteristics when they study the effectiveness

of software-testing techniques. An experiment methodology is presented

that enables experimenters to compare the effectiveness of different

techniques and, at the same time, to measure the influence of defect

characteristics and other factors on the results. The methodology is

demonstrated in a large experiment in the domain of

graphical-user-interface testing.

As the experiment shows, researchers who use the methodology will

understand what kinds of defects tend to be detected by testing and

what testing techniques are better at detecting certain kinds of

defects. This information can help researchers develop more effective

testing techniques, and it can help software testers make better

choices about the testing techniques to use on their projects. As this

work explains, it also has the potential to help testers detect more

defects, and more important defects, during regression testing.