The Problem of Replication
CLICK TO READ MORE FROM:
As intractable as those issues may seem, there are compelling reasons to address them. Pharmaceutical and biotechnology companies depend on academic research when developing new drugs, and erroneous studies waste time and money. “Not being able to rely on research results has made early-stage investing harder,” says Booth, who is an advisor to the Reproducibility Initiative, a network launched by Iorns and other scientists to help researchers independently validate study findings. The National Institutes of Health, meanwhile, has established pilot programs to address replication problems, and some leading science journals are raising the bar on their standards for publication. “Everyone is asking whether this is something we can fix, but it’s clear there are no simple answers,” Fang says.
Repeated experimentation has always been a foundation of scientific discovery. In the 17th century, Robert Boyle, considered the first modern chemist, argued that if findings were to be credible and reliable, they had to be based on methods that independent researchers could learn, assess and replicate. Three centuries later, Austrian philosopher Karl Popper, writing in The Logic of Scientific Discovery in 1934, asserted that “non-reproducible single occurrences are of no significance to science.”
Yet while few may question the importance of replication, the technology and complexity of scientific experimentation today can make it enormously challenging. “A lot of techniques in my laboratory take a long time to master, and there’s a steep learning curve before we can reproduce even our own results,” says Fang, a microbiologist. “So another lab saying ‘We’re going to repeat the high-energy UV laser footprinting you just did on those nucleoprotein complexes’ is going to find it very daunting—and that’s just one component of the experiment.”
But the growing complexity of research methodologies is hardly the only reason replication is no longer a routine part of scientific discovery. “A big factor is that scientists have strong incentives to introduce new ideas, but weak ones to confirm the validity of old ideas,” says Brian Nosek, a psychologist at the University of Virginia. “Innovative findings produce rewards of publication, employment and tenure. Replicated findings produce a shrug.”
In fiscal year 2012, the NIH’s reported annual research funding of $31 billion was down by about 17 percent (adjusted for inflation) from its high in 2003. The number of applicants for NIH grants has soared almost threefold, and the NIH is able to fund fewer than one in five grant proposals. New PhDs must compete for both research dollars and tenure, while senior researchers worry about being able to do the work necessary to extend their careers.
Meanwhile, there may be inadequate training for the postdoctoral students who often play key research roles. And while outright fraud may be rare, it appears to be on the increase. As a percentage of all scientific articles published from January 1973 through May 2012, retractions for fraud or suspected fraud increased tenfold, according to a study Fang and his colleagues published in Proceedings of the National Academy of Sciences in October 2012. “Overt dishonesty is the extreme,” Fang says. “The broad problems of reproducibility have more to do with how the work is presented and how rigorously it has been obtained because of time pressures and the importance of getting positive results.”