For Americans, especially the better-educated, the word of scientists is final. We live our lives on the basis of scientific findings reported in the news media. Social sciences and psychology in particular touch our lives on a daily basis. Relationship columns marshal evidence from “sexologists” and behavioral scientists. Findings about food and nutrition grab our attention. And health publications sell weight-loss tips with the air of credibility a scientist’s nod imparts. But how reliable is science, especially when it’s become a pop profession with so much money involved? Are we building our lives on a sure foundation, or the shifting sands of human error?
The last few years don’t bode well for the reliability of science, at least in key fields. Psychologist Diederik Stapel forced journals to retract 50 papers when the work they were based on — his work — turned out to be fabricated. The Journal of Personality and Social Psychology received a lashing from skeptics after it published a study claiming to support extra-sensory, psychic perception. Eighty-five nonsense papers generated by pranksters at MIT showed up in the proceedings of peer-reviewed computer science conferences, leading to a slew of embarrassing retractions. Even the prestigious journal Science had to pull a political science paper by University of California about the effect of door-to-door canvasing on gay marriage support, after it turned out the researcher had faked his data. And it’s not just sensational stories. The New York Times reports that science paper retractions are rising sharply across the board.
Now a large-scale attempt to reproduce the findings of 100 key psychology studies has returned awkward results. Writing for the Times, Benedict Carey reports that the years-long effort, dubbed “The Reproducibility Project,” found discrepancies in over half of selected studies, confirming “the worst fears of scientists who have long worried that the field needed a strong correction.”
The disturbing findings were reported in full by Science, where the University of Virginia-led project detailed its attempts to replicate the results of studies from 2008 to the present. Despite putting more than 250 researchers to work, the outcome was abysmal for the reliability of peer-reviewed science.
“I think we knew or suspected that the literature had problems,” said Jelte Wicherts of Tilburg University in the Netherlands. “But to see it so clearly, on such a large scale — it’s unprecedented.”
Carey says the revelation could “sow doubt” about the scientific basis for much of what goes on in sociology, psychology, and even medicine.
Many, he writes, have pointed to “a hypercompetitive culture across science that favors novel, sexy results,” This culture “provides little incentive for researchers to replicate the findings of others, or for journals to publish studies that fail to find a splashy result.”
What’s at the root of this toxic culture? Carey suggests the very undertaking of serious peer-review has become “divisive.” Senior researchers “resent the idea that an outsider, typically a younger scientist, with less expertise, would critique the work that often has taken years of study.” For experienced researchers, admits Norbert Schwarz, a psychology professor at the University of Southern California, having other researchers check their work is often perceived as “an attack, a vigilante exercise.”
But perhaps most tellingly, the Replicability Project found that prestigious research groups and universities were no more likely to have produced accurate data than little-known ones. The irony, notes Carey, is that the scientists who pride themselves most on insightfulness and accuracy were every bit as error-prone as their peers.
For those who stake so much of their thinking on the kind of pop-science reported in journals and news magazines, it’s a serious blow. And it’s not limited only to the fields of psychology and social science. The enterprise of science itself has suffered from this failure of accountability, undermining one of the pillars of certainty undergirding modern life.
It’s a rude awakening to the scope of human fallibility, and a reminder of how pride can cripple even the most high-minded undertakings. Rigorous peer-review and fact-checking may be uncomfortable, but they’re indispensable mirrors of the high bar set by God in Scripture.
“A single witness shall not suffice,” writes Moses in Israel’s civil code. “Only on the evidence of two witnesses or of three witnesses shall a charge be established” (Deuteronomy 19:15).
It’s a model of judicial “peer-review” reaffirmed in Proverbs, where the author urges his readers not to trust in themselves alone, but to subject their decisions to scrutiny:
“Without counsel plans fail, but with many advisers they succeed” (Proverbs 15:22).
The future reputation of science may depend on scientists’ willingness to reform their methods and to admit that they’re not infallible. But it’s also a chance for Christians to remind our science-minded friends that no matter how thorough the peer-review process, lasting certainty can’t come from experiments.
Further Reading: