With researchers under financial pressure to come up with cures and breakthroughs, many think that there is too much temptation to produce over-optimistic findings, writes WILLIAM REVILLE
MOST CONCLUSIONS drawn from medical research are misleading, exaggerated or simply wrong. That’s the opinion of Dr John Ioannidis, a physician-researcher at the University of Ioannina, Greece.
Ioannidis claims that up to 90 per cent of published medical research that doctors rely on when treating patients is flawed. His position was summarised by David Freedman in The Atlantic(November 2010).
It seems to me that Ioannidis’s analysis, although perhaps drawing overly harsh conclusions, has also produced some valuable insights. In addition to his medical training Ioannidis is a talented mathematician. One of his research methods is “meta analysis”, whereby hundreds of studies and their predictions are summarised and compared with later real outcomes, eg a new drug is discovered as a cure for a certain disease – does this drug actually prove to be effective over succeeding years?
Research is expensive and scientists fund it by competing for grants that provide funding for two to three years. Research productivity is measured in terms of numbers and quality of research papers published. The most prestigious journals reject up to 90 per cent of papers submitted to them. A successful scientific career depends on regularly winning research grants and publishing papers in prestigious journals. This is a high-pressure game studded with temptations to cut corners.
For example, researchers can become overly committed to hypotheses they use to make their name, even though they have since discovered that other approaches are more reasonable. Researchers can be biased in the types of question they ask, knowing that they are more likely to yield answers they like.
Scientists can be tempted to select research data that favours their hypotheses and to ignore awkward data. Under severe pressure, scientists can be tempted to massage data or even to fabricate data.
In 2005, Ioannidis published two papers that shook the medical research community – one in the online journal PLoS Medicine, and the other in the Journal of the American Medical Association.
In the PLoSarticle Ioannidis proved mathematically that, when it is assumed that there is a modest researcher bias, commonly imperfect research techniques and a common predisposition to concentrate on exciting rather than plausible hypotheses, research findings will point in the wrong direction most of the time.
The PLoSpaper predicted that 80 per cent of non-randomised studies (the most common type), 25 per cent of "gold standard" randomised trials, and 10 per cent of "platinum-standard" large randomised trials must give wrong results. This mathematical prediction corresponds to the rates at which new findings are later refuted. In his JAMApaper, Ioannidis focused on 49 of the most significant medical research findings over the previous 13 years. Forty five of these 49 claimed to have discovered effective new medical treatments. Subsequently, 41 per cent of these claims were shown to be wrong or greatly exaggerated.
Ioannidis is particularly dismissive of the value of nutritional studies, declaring that the methods used in most of these studies are naïve and not nearly discriminating enough to yield meaningful results.
The situation gets even more complicated with the discovery/development of new drugs. Now, not only is individual scientific career progression at stake but big money is involved.
The multinational pharmaceutical companies are a vital part of modern medicine and make great contributions, eg the new drugs that have successfully contained AIDS were largely due to investment in R&D of multinational “big pharma”. These companies must make money.The temptation is obvious if you have spent years of work and hundreds of millions of dollars developing a new drug, and then new research shows that it is not effective. Usually the company will bite the bullet and take the hit, but occasionally . . .
Medical journals and popular accounts of medical research are full of predictions that massive breakthroughs are imminent.
A good current example is the optimistic predictions about the many cures that will spring from embryonic stem cell research. Most of this is fuelled by the internal dynamics of the way we do science. Actual breakthroughs come slowly enough.
In 1970 President Richard Nixon declared a war on cancer. It was expected that the entire problem would succumb to scientific onslaught within 10 years or so. Over 40 years later you see where we stand.
The way we organise scientific research greedily demands quick success. This tends to inhibit scientists from admitting to mistakes and even encourages them to disguise mistakes as successes. Science is effective. It remains the most successful adventure ever of the human spirit, but it works relatively slowly. Ioannidis is of the opinion that we must learn to be comfortable with that fact. I agree with him.
William Reville is Professor of Biochemistry and Public Awareness of Science Officer at UCC. understandingscience.ucc.ie
Ioannidis is particularly dismissive of the value of nutritional studies