How Well Do My Results Generalize? Comparing Security and Privacy Survey Results from MTurk and Web Panels to the U.S.

Abstract

Security and privacy researchers often rely on data collected from Amazon Mechanical Turk (MTurk) to evaluate security tools, to understand users' privacy preferences, to measure online behavior, and for other studies. While the demographics of MTurk are broader than some other options, researchers have also recently begun to use census-representative web-panels to sample respondents with more representative demographics. Yet, we know little about whether security and privacy results from either of these data sources generalize to a broader population. In this paper, we compare the results of a survey about security and privacy knowledge, experiences, advice, and internet behavior distributed using MTurk (n=480), a nearly census-representative web-panel (n=428), and a probabilistic telephone sample (n=3,000) statistically weighted to be accurate within 2.7% of the true prevalence in the U.S. Surprisingly, we find that MTurk responses are slightly more representative of the U.S. population than are responses from the census-representative panel, except for users who hold no more than a high-school diploma or who are 50 years of age or older. Further, we find that statistical weighting of MTurk responses to balance demographics does not significantly improve generalizability. This leads us to hypothesize that differences between MTurkers and the general public are due not to demographics, but to differences in factors such as internet skill. Overall, our findings offer tempered encouragement for researchers using MTurk samples and enhance our ability to appropriately contextualize and interpret the results of crowdsourced security and privacy research.