In 2005, John Ioannidis, a professor of medicine at Stanford University, published a paper, “Why most published research findings are false,” mathematically showing that a huge number of published papers must be incorrect. He also looked at a number of well-regarded medical research findings, and found that, of 34 that had been retested, 41% had been contradicted or found to be significantly exaggerated.
Since then, researchers in several scientific areas have consistently struggled to reproduce major results of prominent studies. By some estimates, at least 51%—and as much as 89%—of published papers are based on studies and experiments showing results that cannot be reproduced.
Researchers have recreated prominent studies from several scientific fields and come up with wildly different results. And psychology has become something of a poster child for the “reproducibility crisis” since Brian Nosek, a psychology professor at the University of Virginia, coordinated a Reproducibility Initiative project to repeat 100 psychological experiments, and could only successfully replicate 40%.
Now, an attempt to replicate another key psychological concept (ego depletion: the idea that willpower is finite and can be worn down with overuse) has come up short. Martin Hagger, psychology professor at Curtin University in Australia, led researchers from 24 labs in trying to recreate a key effect, but found nothing. Their findings are due to be published in Perspectives on Psychological Science in the coming weeks.
Why are they getting it wrong?
No one is accusing the psychologists behind the initial experiments of intentionally manipulating their results. But some of them may have been tripped up by one or more of the various aspects of academic science that inadvertently encourage bias.
For example, there’s massive academic pressure to publish in journals, and these journals tend to publish exciting studies that show strong results.
“Journals favor novelty, originality, and verification of hypotheses over robustness, stringency of method, reproducibility, and falsifiability,” Hagger tells Quartz. “Therefore researchers have been driven to finding significant effects, finding things that are novel, testing them on relatively small samples.”
This has created a publication bias, where studies that show strong, positive results get published, while similar studies that come up with no significant effects sit at the bottom of researchers’ drawers.
Meanwhile, in cases where researchers have access to large amounts of data, there’s a dangerous tendency to hunt for significant correlations. Researchers can thus convince themselves that they’ve spotted a meaningful connection, when in fact such connections are totally random.