The Problem of Replication

Massachusetts General
  • Linda Keslar

Complicating debates about the reasons for low rates of replication is uncertainty about the magnitude of the problem. In a 2005 essay in Public Library of Science (PLOS) Medicine, John Ioannidis, an epidemiologist and professor at Stanford School of Medicine, argued that most published research findings are false, and used statistical models to underscore issues with how studies are conceived and designed. In 2009, Ioannidis and colleagues zeroed in on the repeatability of 18 studies of gene expression published in Nature Genetics in 2005 and 2006. Insufficient data made replication impossible for 16 of the papers.

In 2011, German pharmaceutical company Bayer HealthCare reported in the journal Nature Reviews that its scientists had been unable to reproduce nearly three-quarters of 67 published studies in cardiovascular disease, cancer and women’s health. In most cases, the inability to replicate results led to the termination of research efforts, a trend that may help explain why success rates for clinical drug trials have been declining. “Bayer HealthCare has become more cautious when working with published research targets,” says Khusru Asadullah, head of global biomarkers at Bayer’s Berlin headquarters and an author of the Nature Reviews article. “Targets now have to be better validated internally before we start big projects.”

Last year, Lee Ellis of MD Anderson and C. Glenn Begley, former head of global cancer research at pharmaceutical company Amgen, chronicled in the journal Nature how Amgen scientists attempted to replicate 53 landmark cancer studies and found that they could confirm only six. The scientists even consulted with the original investigators, who in some cases were unable to repeat their own experiments. But because Amgen investigators were bound by confidentiality agreements, the paper left many unanswered questions. “They didn’t reveal a list of which studies they couldn’t reproduce,” Fang says.

Begley, now chief scientific officer at TetraLogic Pharmaceuticals, has since provided more details, and he published his analysis, “Six Red Flags for Suspect Work,” in Nature in 2013. “If researchers got the results they liked in the first experiment, they usually didn’t repeat it,” Begley says. Much of today’s research isn’t fudged, he says, or fraudulent: “It’s lazy and sloppy.”

New research by Ellis and a team at MD Anderson published in PLOS ONE in 2013 provided yet another perspective on the reproducibility problem. They reported that half of more than 400 respondents at the institution said they had been unable to replicate at least one published study. Seventy-eight percent of the scientists had attempted to contact the authors of the original scientific paper to identify the problem, but only one-third received a helpful response. More than 40 percent reported difficulties finding an outlet to publish findings that contradicted previous results. Such problems increase the likelihood that “suspect findings may lead to the development of entire drug development or biomarker programs that are doomed to fail,” the authors wrote.

(continued)

More From Massachusetts General >>
Read Next Article >>
Got grapes? UCLA researchers have demonstrated how resveratrol, an antioxidant derived from grapes...
UCLA has been tapped by the Defense Advanced Research Projects Agency to spearhead an innovative...
Harry Campbell
Neurologist Bruce Miller calls tau—a floppy, free-form protein—“the holy grail of dementia.” That...
Physicians have long assumed that ovarian cancer—one of the deadliest cancers among women—...
Wrong-level spine surgeries—when a surgeon accidentally operates on the wrong vertebra—occur with...
Photo by Sam Kaplan; Copyright Sam Kaplan
In 2012, when the journal Science published a study by a group of Cleveland researchers touting a...
Photo by Movus; http://www.gettyimages.com
Patients who come to Johns Hopkins for epilepsy surgery sometimes require a prequel to their...
Copyright Getty Images
In the late 1990s, the process of isolating genes from the chromosomes in which they reside was...
Photo by Sergey Anatolievich Pristyazhnyuk; copyright Getty Images
The human body has many natural barriers intended to prevent microorganisms, diseases, and toxins...
Illustration by Brian Stauffer
 When Fred Lesikar arrived at the hospital in October 2009 suffering sharp pain in his upper back,...