DISTRIBUTED CONTINUOUS QUALITY ASSURANCE
Abstract
Quality assurance (QA) tasks, such as testing, profiling, and
performance evaluation, have historically been done in-house on
developer-generated workloads and regression suites. The shortcomings
of in-house QA
efforts are well-known and severe, including (1) increased
QA cost and (2) misleading results when the test cases,
input workloads, and software platforms at the developer's site
differ from those in the field. Consequently, tools and processes have
been
developed to improve software quality by increasing user participation
in the QA process. A limitation of these approaches is that they
focus on isolated mechanisms, not on the coordination and control
policies and tools needed to make the global QA process efficient,
effective, and scalable. To address these issues, we have initiated
the Skoll project, which is developing and validating novel software
QA processes and tools that leverage the extensive computing resources
of worldwide user communities in a distributed, continuous manner to
significantly and rapidly improve software quality. We call this
distributed, continuous quality assurance (DCQA). We envision a QA
process conducted around-the-world, around-the-clock on a powerful
computing grid provided by thousands of user machines during
off-peak hours. Skoll processes are distributed, opportunistic, and
adaptive. They are distributed: Given a QA task we divide it into
several subtasks each of which can be performed on a single user
machine. They are opportunistic: When a user machine becomes available
we allocate one or more subtasks to it, collect the results when
they are available, and fuse them together at central collection sites
to complete the overall QA process. And finally they are adaptive: We
use earlier subtask results to schedule and coordinate subtask
allocation.
In this thesis,
we build an infrastructure, algorithms and
tools for developing and executing through, transparent, managed, and
adaptive DCQA processes.
We then develop several novel DCQA processes
and empirically evaluate them, with a special focus on cost
efficiency and applicability of these processes to real-life,
highly-configurable software systems. Our results strongly suggest
that these new processes are an effective and efficient way to conduct
QA tasks such as evaluating performance characteristics and testing for
functional correctness of evolving software systems.