Exact replication studies may accelerate development of better cancer treatments

Mar 15 2017

In response to rising concern that many published scientific results may be false, the Reproducibility Project: Cancer Biology set out to replicate findings from the 50 most cited cancer studies from 2010-2012. A perspective in AACC's Clinical Chemistry journal discusses the project's preliminary results and suggests changes the research community can make to prevent reproducibility issues from inhibiting cancer care.

The translation of cancer research into real-world treatments is strikingly inefficient, with only 5% of potential cancer drugs making it from the research lab through phase III clinical testing, compared with the 20% of cardiovascular therapies that pass phase III testing. One potential explanation for this high failure rate is that very few cancer research findings are confirmed by independent studies. Recently, the major pharmaceutical company Amgen was only able to validate 6 out of 53—or 11% of—landmark cancer papers, but did not release the list of studies checked due to confidentiality issues. The Reproducibility Project: Cancer Biology was launched in an effort to build on this investigation in a more transparent manner.

In January, the Reproducibility Project released the results of its first five replication studies, one of which failed outright, while the other four yielded inconclusive or moderately successful results. In this perspective, John P.A. Ioannidis, MD, DSc, of Stanford University in Stanford, California—an expert on reproducibility—discusses how the failed replication study underscores the limitations of conceptual replication, which is the current standard for confirming biomedical research results.

Prior to the Reproducibility Project—which used exact replication—at least 10 other labs had seemingly validated the unreplicated paper's findings that a particular peptide can enhance the efficacy of chemotherapy drugs. However, Ioannidis notes that these validation studies were conceptual replications instead of exact, which means their experimental conditions differed from those in the original study. For example, the validating labs looked at different cancer types or different anti-tumor peptides. Additionally, it is unknown whether these other labs tried multiple variants on the original study before finding one that yielded the desired results, since this information isn't documented with conceptual replications.

Unfortunately, it is very difficult for researchers to publish exact replication studies because they aren't novel, which is why conceptual replications proliferate. As this example from the Reproducibility Project shows though, encouraging and funding exact replication is essential to confirming cancer research results and accelerating the development of cancer treatments that will benefit patients.

"For those who question how much we can spend on replication, the answer is probably a thousand-fold more, easily," said Ioannidis. "The entire [Reproducibility Project: Cancer Biology] costs $2 million. Waste in biomedical research is estimated to be tens, perhaps even hundreds of billions, annually. To do better, insights on reproducibility will be crucial. It is not about shaming and tarnishing reputations. It is about whether our observations are solid and eventually can have practical value."

Source: https://www.aacc.org/