Building Knowledge through Families of Software Studies: An Experience Report

View/ Open
Date
1999-02-04Author
Basili, Victor
Shull, Forrest
Lanubile, Filippo
Metadata
Show full item recordAbstract
Experimentation in software engineering is difficult. One reason is that
there are a large number of context variables, and so creating a
cohesive understanding of experimental results requires a mechanism for
motivating studies and integrating results. It requires a community of
researchers that can replicate studies, vary context variables, and
build abstract models that represent the common observations about the
discipline. This paper discusses the experience of the authors, based
upon a collection of experiments, in terms of a high level framework for
organizing sets of related studies. With such a framework, experiments
can be viewed as part of common families of studies, rather than being
isolated events. Common families of studies can contribute to higher
level hypotheses that no individual experiment could achieve. Then the
replication of experiments within a family of studies can act as the
cornerstone for building knowledge in an incremental manner. A mechanism
is suggested that motivates, records, and integrates individual
experiments within a family for analysis by the community at large.
To support the framework, this paper discusses the experiences of the
authors in carrying out empirical studies, with specific emphasis on
persistent problems encountered in experimental design, threats to
validity, criteria for evaluation, and execution of experiments in the
domain of software engineering. (Also cross-referenced as
UMIACS-TR-99-05)