The Generalizability of Survey Experiments (WP-14-19)


IPR-WP-14-19

Kevin Mullinix, Thomas J. Leeper, James Druckman, and Jeremy Freese

Experiments embedded in surveys have become a central methodology across the social sciences and, in particular, political science. Researchers can combine experiments’ causal power with the generalizability of population-based samples. Yet, the expense and difficulty of employing population samples has led many to turn to online convenience samples such as the crowdsourcing platform Amazon Mechanical Turk (MTurk). This has reinvigorated debates about the external validity of convenience samples in experiments. Mullinix, Druckman, and Freese identify and test conditions where these inexpensive convenience samples provide experimental inferences similar to those of more costly population samples. Results from twenty experiments implemented on both types of samples show that, as predicted, results are often highly similar and are predictable in their divergence. Consequently, social scientists can often draw generalizable causal inferences using convenience samples that are less than 5 percent of the cost of population-based samples.

Kevin Mullinix, Assistant Professor, Department of Government and Justice Studies, Appalachian State University

Thomas J. Leeper, Professor in Political Behaviour, London School of Economics and Political Science

James Druckman, Payson S. Wild Professor of Political Science, Associate Director and Faculty Fellow, Institute for Policy Research, Northwestern University

Jeremy Freese, Professor of Sociology, Stanford University

Download the published paper

PDF icon Download working paper PDF