Don’t blame journalists for overhyping science – it’s often the fault of scientists themselves
That many scientific fields are rife with error, bias and outright fraud may be unsurprising given the succession of news stories about prominent psychologists faking results and pop-psychology gurus fooling gullible Conservative politicians into ‘power posing’ at party conferences. But Stuart Ritchie’s Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science, published this month, demonstrates that similar problems exist throughout virtually all the sciences, including ‘hard’ sciences such as chemistry or medicine that can end up with deadly consequences.
Psychology, Ritchie’s own field, has been particularly badly hit. A so-called ‘replication crisis’ has hit the field, with psychologists unable to repeat the results of key experiments, including a famous one on the ‘priming effect’, where researchers were apparently able to influence people’s behaviour simply by showing them the right words – subjects shown words associated with elderly people (old, grey, wise, knits, and Florida) were reported as having walked more slowly out of the lab than the control group. The Stanford Prison Experiment, the well-known role-play experiment which purportedly showed people’s willingness to abuse power, was recently exposed as being virtually fraudulent, with its designer egging on the guards to act into their roles, rendering the results of the whole exercise useless.
Studies like these formed the basis for some of the most hyped-up social science of the 21st century. Two of the most popular and influential books of the past twenty years, Thinking, Fast and Slow by Nobelist Daniel Kahneman and Nudge by Cass Sunstein and Nobelist Richard Thaler, both cite dozens of papers that either failed to replicate or should have been too small to count as real evidence in the first place. The fact that behavioural economics sits on a throne of lies – weak, unreplicable or even downright fraudulent research – has not yet reached many policymakers, who still think of things like priming effects and ‘choice architecture’ as the hottest show in town.
But psychology, by now, is a soft target. Ritchie’s four cardinal sins – fraud, bias, negligence and hype – are found across the sciences. Fraud is staggeringly common: of all papers retracted for any reason, more than half are retracted for consciously unethical behaviour like fraud or plagiarism, rather than because some accidental error has been found. Bias – which leads researchers to cherry-pick their data and contort their hypotheses to generate interesting results – is such a problem that only 13% of studies in the five highest-impact medical journals reported everything they had said they would in advance.
Negligence with things like data and samples has led errors to pervade entire fields. One example is the study of cell lines, cultures of animal cells that can be used in place of primary cells for biological and medical research, which are essentially immortal and can thus be the basis for experiments over long periods of time. Here, mislabeled samples and cluttered labs have led to thousands of errors, with scientists thinking they were working with, say, human bone cancer cells when they were actually working with cells from a pigs’ colon. This problem has been so widespread that a 2017 analysis found that a whopping 32,755 studies had been published using so-called imposter cells, and over half a million further studies published that cited those misidentified experiments.
The most surprising thing about Ritchie’s fourth sin, hype, is that it is the scientists themselves – not journalists – who are often responsible. These scientists are frequently heavily involved in writing the press releases about their own papers, and researchers have found that if the press release hyped up a finding there was a much greater chance of the media exaggerating them too. Press releases that gave advice (for example, about what foods to avoid to lower your risk of cancer), that made causal claims about a correlational finding, or that reported findings generated in lab rats as having clear implications for humans were vastly more likely to be ‘sexed up’ in the media than press releases that were more circumspect. For once, it is not the journalists who are to blame.
As Ritchie is a friend of mine, I will not commit one of the sins he identifies (scientists referring their papers to their friends for peer review) by trying to review the book neutrally. In my own, biased opinion, it is an enjoyable, informative and important book about the problems faced by modern science. As well as the book’s devastating assessment of the sciences, it is extremely useful: Ritchie takes the time to clearly explain the concepts and jargon used in debates about scientific accuracy so that any intelligent layperson can understand them – p-values, statistical power, h-indexes – and lays out with clear, simple diagrams how to visualise and understand problems like p-hacking and publication bias. The book even includes a short appendix, ‘How to read a scientific paper’, that should be required reading for every student and media pundit.
But the problems Ritchie identifies require more than people simply understanding them. They are endemic to the whole structure of science, driven in large part, he says, by the ‘publish or perish’ culture in academia that gives even well-intentioned, truth-seeking scientists incentives to cook the books.