Demystifying Property-Based Testing through Evaluation & Visualization

Abstract

Property-based testing (PBT) is a testing methodology where programmers define properties, i.e., executable specifications, that are validated against a large volume of randomized inputs. While PBT’s underlying theory is simple and accessible, the learning curve for writing and interpreting meaningful property-based tests remains steep. This difficulty arises from several factors, including confusion caused by the wide variety of PBT frameworks across different programming languages, limited understanding of available input generation strategies, and insufficient or hard-to-access information about generated inputs and test outcomes.

In this thesis, I aim to make PBT more approachable and effective for programmers by extending two platforms with additional PBT frameworks: ETNA, to evaluate and compare different input generation strategies across various PBT frameworks, and Tyche, to provide an interactive and configurable user interface to make sense of the results of the property-based tests.

To evaluate how ETNA and Tyche aid users in developing more meaningful property-based tests, I integrate a workload, the simply typed lambda calculus (STLC), from the ETNA platform with Tyche to the two frameworks that I extended Tyche with: QCheck and Rackcheck. In the workloads, I analyze the expressions generated with Tyche, and tune the generators accordingly to improve performance.