The Generalizability of Survey Experiments (WP-14-19)
Kevin Mullinix, Thomas J. Leeper, James Druckman, and Jeremy Freese
Experiments embedded in surveys have become a central methodology across the social sciences and, in particular, political science. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, the expense and difficulty of employing population samples has led many to turn to online convenience samples such as the crowdsourcing platform Amazon Mechanical Turk (MTurk). This has reinvigorated debates about the external validity of convenience samples in experiments. Mullinix, Druckman, and Freese identify and test conditions where these inexpensive convenience samples provide experimental inferences similar to those of more costly population samples. Results from twenty experiments implemented on both types of samples show that, as predicted, results are often highly similar and are predictable in their divergence. Consequently, social scientists can often draw generalizable causal inferences using convenience samples that are less than 5 percent of the cost of population-based samples.