How Well Do My Results Generalize? Comparing Security and Privacy Survey Results from MTurk and Web Panels to the U.S.

Redmiles, Elissa M.; Kross, Sean; Pradhan, Alisha; Mazurek, Michelle L.

How Well Do My Results Generalize? Comparing Security and Privacy Survey Results from MTurk and Web Panels to the U.S.

dc.contributor.author	Redmiles, Elissa M.
dc.contributor.author	Kross, Sean
dc.contributor.author	Pradhan, Alisha
dc.contributor.author	Mazurek, Michelle L.
dc.date.accessioned	2017-02-22T00:01:51Z
dc.date.available	2017-02-22T00:01:51Z
dc.date.issued	2017-02-21
dc.description.abstract	Security and privacy researchers often rely on data collected from Amazon Mechanical Turk (MTurk) to evaluate security tools, to understand users' privacy preferences, to measure online behavior, and for other studies. While the demographics of MTurk are broader than some other options, researchers have also recently begun to use census-representative web-panels to sample respondents with more representative demographics. Yet, we know little about whether security and privacy results from either of these data sources generalize to a broader population. In this paper, we compare the results of a survey about security and privacy knowledge, experiences, advice, and internet behavior distributed using MTurk (n=480), a nearly census-representative web-panel (n=428), and a probabilistic telephone sample (n=3,000) statistically weighted to be accurate within 2.7% of the true prevalence in the U.S. Surprisingly, we find that MTurk responses are slightly more representative of the U.S. population than are responses from the census-representative panel, except for users who hold no more than a high-school diploma or who are 50 years of age or older. Further, we find that statistical weighting of MTurk responses to balance demographics does not significantly improve generalizability. This leads us to hypothesize that differences between MTurkers and the general public are due not to demographics, but to differences in factors such as internet skill. Overall, our findings offer tempered encouragement for researchers using MTurk samples and enhance our ability to appropriately contextualize and interpret the results of crowdsourced security and privacy research.	en_US
dc.identifier	https://doi.org/10.13016/M2BV8D
dc.identifier.uri	http://hdl.handle.net/1903/19164
dc.language.iso	en_US	en_US
dc.relation.ispartofseries	UM Computer Science Department;CS-TR-5054
dc.title	How Well Do My Results Generalize? Comparing Security and Privacy Survey Results from MTurk and Web Panels to the U.S.	en_US
dc.type	Technical Report	en_US