Research News

Are Most Published Research Findings False?

IPR associate Eli Finkel finds the anti-false-positives movement has “overreached”


journals
Journals and researchers seek to publish statistically significant results, but are most published findings false?

Over the past decade, articles in mainstream publications (The New Yorker) and scientific journals (Perspectives on Psychological Science) have asserted that science is in crisis—and even, as Stanford medicine professor John Ioannidis famously asserted in PLoS Medicine, that “most published research findings are false.”

Since the articles that make a researcher’s career and boost a journal’s popularity feature studies with statistically significant results, this can motivate researchers to tweak their data, thereby creating “false positives” in the scientific literature—that is, instances when researchers conclude that an effect is true, when it is in fact false. Hence, the assumption that research is rife with false conclusions.

The Anti-False-Positives (AFP) movement arose in response to this issue and demands more stringent guidelines for researchers and journals to minimize false positives in scientific data. But in their race to rectify the issue, the AFP movement has “overreached,” social psychologist and IPR associate Eli Finkel and his colleagues—the University of Texas at Austin’s Paul Eastwick and Harry Reis of the University of Rochester—argue in a new article.

Issues surrounding false positives “are important for us to be discussing, but there’s a chance that the correctives, if insufficiently thought-out, can do more damage than good,” Finkel explained. “In particular, as the rules for publication get increasingly stringent, we will exacerbate an equally important type of error called ‘false negatives’—instances when researchers conclude that an effect is false, when it is in fact true.”

Eli Finkel
Eli Finkel

The recommendations offered by the AFP movement are likely to produce these problems when scholars are pursuing particularly ambitious projects, such as the General Social Survey, which started collecting data in 1972.

“If you’re talking about psychology laboratory experiments, for example, you can preregister your precise methods and hypotheses every academic term,” Finkel said. “But if you’re talking about ambitious, longitudinal data sets, some of the AFP recommendations are not practical—or even sensible.”

Finkel and his colleagues offer guidance on how both researchers and AFP supporters can work within the AFP movement’s recommendations for reducing false positives, which include prepublication sharing of materials, postpublication sharing of data, close replication, and avoiding piecemeal publication, in addition to preregistering studies and increasing sample size. For instance, in the case of preregistration, they recommend that scientists preregister the theories and data analyses they wish to perform on longitudinal data, even if they cannot register the data themselves; AFP adherents, on the other hand, should demonstrate flexibility by accepting and trusting preregistrations in this form.

Rather than assuming that most published research findings are false or implementing new policies with great haste, “We should view this crisis as an opportunity to evaluate our practices—and, once we’ve done that, updating policies as needed,” Finkel said.

Eli Finkel is a professor in psychology and in the Kellogg School of Management, and an IPR associate. The article, “Best Research Practices in Psychology: Illustrating Epistemological and Pragmatic Considerations with the Case of Relationship Science,” was published in the Journal of Personality and Social Psychology.

Photo Credit: Tobias von der Haar