Why Replications Do Not Fix the Reproducibility Crisis: A Model and Evidence from a Large-Scale Vignette Experiment (WP-19-04)


WP-19-04

Adam J. Berinksy, James N. Druckman, Teppei Yamamoto

Scientists have become increasingly concerned that “most published research findings are false” Ioannidis
(2005), and have emphasized the need for replication studies. Replication entails a researcher repeating a
prior research study with newly collected data. The mixed results of large-scale replication efforts have led
some to conclude there is a “reproducibility crisis”: false positives are pervasive. One solution is to
encourage more replications. Yet, replication studies can alter the published literature only if they actually
are published. And it may well be that replication studies themselves are subject to “publication bias.” The
researchers offer a micro-level model of the publication process involving an initial study and a
replication. The model incorporates possible publication bias both at the initial and replication stages. This
enables them to investigate the implications of publication biases on various statistical metrics of evidence
quality. They then estimate the key parameters of the model with a large-scale vignette experiment
conducted with political science professors teaching at Ph.D.-granting institutions in the United States.
Their results show substantial evidence of publication bias: on average, respondents judged statistically
significant results about 20 percentage points more likely to be published than statistically insignificant
results. They further find evidence of what they call a “gotcha bias.” Replication studies that run contrary
to the existing literature are more likely to be published than those consistent with past research.
Publication biases at the replication stage also can lead to the appearance of increased reproducibility even
when there are actually more false positive results entering the published literature.

Adam J. Berinksy, Professor of Political Science, Massachusetts Institute of Technology

James N. Druckman, Payson S. Wild Professor of Political Science and IPR Fellow, Northwestern University

Teppei Yamamoto, Associate Professor of Political Science, Massachusetts Institute of Technology

 

PDF icon Download working paper PDF