Research News

Combatting the Credibility Crisis in Research

IPR sociologist Jeremy Freese traces the rise of forensic objectivity


steak
One day red meat is good for you; the next it is not. Open science, part of the forensic objectivity movement, could make the findings of nutrition, and other, studies more transparent.

Recently, the journal Science retracted a widely reported study on attitudes towards same-sex marriages when it came to light that its dataset and findings might have been fabricated. When researchers fudge their data to obtain the statistically significant results that journals tend to publish—or fail to replicate a previous study’s results—they reinforce a prevailing belief that science is experiencing a “crisis of credibility.” 

In response to this crisis, a new scientific movement, “forensic objectivity,” has arisen. IPR sociologist Jeremy Freese and IPR graduate research assistant David Peterson chronicle this movement and its future in their recent IPR working paper.

freese
Jeremy Freese

“Normally, one would think of objectivity in science as being primarily grounded in the methods people use,” Freese said. “But there has been this shift toward trying to think more about using sets of findings to speak about the integrity of literatures,” or meta-analysis.

By looking at all published studies on a certain topic simultaneously and using techniques like funnel plots or p-curves, researchers can determine if the statistical patterns in the studies’ results are unlikely to occur on their own—and might have been manipulated to be statistically significant.

Another part of the forensic objectivity movement involves open science, in which researchers publicly share their datasets, thought processes, and methodologies. For instance, journals have started rewarding scientists for making their raw data available or for explaining their experimental design in advance of data collection.

According to Freese and Peterson, such efforts will not only engender trust among researchers, but also can help bridge the gap between researchers’ conclusions in scientific articles and what they present to the public, for example, in press releases. It is a gap particularly evident in nutrition research, where one study reports one day that it is fine to eat red meat, which is then contradicted by another study on a subsequent day, where it is not. 

“Ultimately, dramatizing findings with the public has this long-run negative effect of leading to unrealistic expectations about science—about what can be learned from an individual study, and ultimately distrust,” Freese said.

The hope of the forensic objectivity movement, the researchers conclude, is that its analyses of scientific research will encourage science as a whole to adjust its incentives, making it easier for researchers to publish journal articles without statistically significant results and to self-police their fields.

Jeremy Freese is Ethel and John Lindgren Professor of Sociology and an IPR fellow