The Problem of Replication

Massachusetts General

But before testing whether the drug could help human Alzheimer’s patients, scientists needed to make sure it worked in mice—and while some labs were able to reproduce the drug’s effect on memory and cognition, the plaque reduction effect couldn’t be replicated. In four technical comments published in Science last May, several teams of independent researchers, including those co-authored by Sisodia and Tanzi, reported that their replication attempts showed no effect on plaque levels in lab mice treated with bexarotene. Months later, in August, a paper in the journal Molecular Neurodegeneration reported that researchers from Johns Hopkins had also failed to replicate either the plaque reduction or memory and cognition effects found in the Case Western research.

For anyone hoping to see progress in the fight against Alzheimer’s, the failure of the follow-up research was disappointing. But it also points to the necessity and challenges of independently validating published research findings. While reproducibility is considered a bedrock of scientific discovery, there has been growing concern about the quality of recent studies. “Data reproducibility means that the seminal findings of a paper can be reproduced in any qualified lab that has appropriate resources and expertise,” says Lee Ellis, a surgeon and researcher at the University of Texas MD Anderson Cancer Center in Houston. “If you try to reproduce all of the findings in a paper, you’re likely to find some divergent outcomes, but the point of the paper should remain the same.”

But Ellis and others who have explored these issues have found that medical research, including seemingly groundbreaking work, is reproducible less than half the time. “The unspoken rule is that at least 50 percent and more like 70 percent of the studies published even in top-tier academic journals can’t be repeated,” says Bruce Booth, a partner at Atlas Venture, a venture capital firm in Boston. “Everyone recognizes reproducibility as a big problem,” says Elizabeth Iorns, a cancer researcher in Palo Alto, Calif., and chief executive of Science Exchange, an online marketplace for scientific resources and expertise.

Many factors contribute to the low odds of reproducibility. The original experiments may have been poorly designed, or there could be problems with how results were analyzed. The trend may also be a symptom of a scientific community in which the job market and funding are tighter than ever, and researchers must publish or perish, leading to a lack of rigor in their research. “It is a dysfunctional scientific climate,” says Ferric Fang, a professor at the University of Washington School of Medicine and editor-in-chief of the journal Infection and Immunity. And because journals favor original research, scientists have little incentive to pursue replicative work.

(continued)

More From Massachusetts General >>
Read Next Article >>
Got grapes? UCLA researchers have demonstrated how resveratrol, an antioxidant derived from grapes...
Photo courtesy of The Life Picture Collection/Getty Images
Medicine’s first superpower—X-ray vision—has led to improved imaging techniques that allow for...
Our bodies may not have caught up with the Bombardier Global 6000, but new research illuminates how...
UCLA has been tapped by the Defense Advanced Research Projects Agency to spearhead an innovative...
Harry Campbell
Neurologist Bruce Miller calls tau—a floppy, free-form protein—“the holy grail of dementia.” That...
Physicians have long assumed that ovarian cancer—one of the deadliest cancers among women—...
Wrong-level spine surgeries—when a surgeon accidentally operates on the wrong vertebra—occur with...
Photo by Sam Kaplan; Copyright Sam Kaplan
In 2012, when the journal Science published a study by a group of Cleveland researchers touting a...
Photo by Movus; http://www.gettyimages.com
Patients who come to Johns Hopkins for epilepsy surgery sometimes require a prequel to their...
Copyright Getty Images
In the late 1990s, the process of isolating genes from the chromosomes in which they reside was...