Designing Effective Property-Based Testing Frameworks

Abstract

Property-based testing (PBT) is a widely adopted random testing methodology thatcombines user-defined logical propositions with automated test case generation.

Over the past two decades, PBT has gained significant traction; today, virtually allmajor programming languages host multiple competing PBT frameworks, each innovating in interface design and underlying techniques.

As a result of this proliferation, the PBT landscape of today is significantlyfragmented. Advances in one framework often fail to reach others. Common tips, tricks, and techniques are rarely evaluated quantitatively. Historical design choices, once built into popular libraries, can keep the ecosystem from adapting. As a result, the field still lacks a shared empirical foundation for deciding which PBT designs work best, when, and why.

This thesis takes a step toward consolidating PBT knowledge. It contributes aconceptual framework for describing property runners in the wild; a flexible property language for expressing runners distributed throughout the PBT ecosystem; ETNA, the first large-scale evaluation and benchmarking platform for PBT; and DIRT, a domain-specific testing approach that clarifies which challenges remain outside the reach of general-purpose consolidation. Together, these contributions make it easier to compare PBT techniques, transfer ideas across frameworks, and build the next generation of testing tools on shared evidence rather than local convention.