Major study shows that most research is wrong

Major study shows that most research is wrong

Written by Debbie Todd
25th October 2010 | 4:34am

7 Comments

Question. Where would you expect to find an article with the title: “Why Most Published Research Findings Are False”? Answer. On a US government website that publishes research findings.

On the website of the National Center for Biotechnology Information (NCBI) there is an article which just might blow your mind. You could be forgiven for imagining that this was some antiscientific mentally challenged individual who had somehow managed to worm their way into the hallowed portals of scientific officialdom in order to spread some crazily hysterical notions.

But you’d be wrong. This is in fact a sober and systematic analysis of research findings. It also makes the seemingly less-than-sensational claim that “Published research findings are sometimes refuted by subsequent evidence”. I suppose it’s the word ‘most’ in the article’s title that provides much more of a shock-factor than that comparatively mild sounding claim.

For the reader (of the this work of seemingly unimpeachable provenance and pedigree by Professor John P. A. Ioannidis, of Tufts University School of Medicine and Ioannina School of Medicine, Greece) the problem is not whether to take the work seriously, but whether it will suffer the poetic ignominy of being itself refuted by subsequent evidence. Personally, I doubt that it will.

Here’s a brief sample. The work is published on the website in its entirety.

“There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims. However, this should not be surprising.

It can be proven that most claimed research findings are false. Here I will examine the key factors that influence this problem and some corollaries thereof”.

Here are his corollaries. I’ve added my comments at the end of each one.

(1) The smaller the studies conducted in a scientific field, the less likely the research findings are to be true. My take: bigger studies often refute smaller ones, frequently setting out to do precisely that.

(2)  The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. My take: small effects are easier to generate than big ones, more likely to be misleading.

(3) The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true. I don’t have a view on this, I suspect that this is a ‘discipline specific’ (i.e. ‘medical research’ in this study’s case) issue.

(4) The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true. Again I’m going to retreat behind a discipline-specific defence for not having a view on this, although I can sense a ‘deviation from best practice’ or ‘operating outside of an established practice regime’ aspect, which of course heightens potential risk.

(5) The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true. I’m not sure this is controversial.

(6) The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. I just love the fact that the term ‘hot’ has appeared in such a serious context. Reminds me of McLuhan, who also did that.

Here’s my analysis in the context of innovation and startups

What this report is really all about is risk. Research studies are all about reducing risk. Just doing something without doing prior research is risky. Doing some research beforehand reduces risk, but it doesn’t eliminate it, because the findings may be wrong. The more risky the thing you want to do, doesn’t mean that the research is necessarily going to be more thorough: it might be aimed at doing the precise opposite, in order to provide less opportunity for discovering a reason not to proceed.

A theory is only as strong as the last unsuccessful attempt to refute it proved it to be. What is it that we are measuring when we consider the strength of a theory by the number of unsuccessful attempts at refutation it has withstood, or the number of years it has withstood refutation? Nobody knows. Professor Karl Popper asked these questions many years ago and nobody is completely satisfied with any answer we have come up with since. So much for refutation. It doesn’t mean as much as you’d imagine, because it’s ultimately no more or less guaranteed than the thing it’s refuting.

So, if most research findings are wrong, should we give up on research, or should we do less research, or just do bigger studies, or should we stick to ‘safer bets’ where the last big unrefuted research findings are more likely to have been right? His answer to this, perhaps surprisingly, seems to be no.  He comments on this in his last paragraph.

Nevertheless, he says “most new discoveries will continue to stem from hypothesis-generating research with low or very low pre-study odds”. In other words (glib conclusion coming) we need to take risks in order to innovate in research just as much as we need to do in enterprise. We need to be prepared to do research that has a good chance of producing findings which are wrong, in order have a good chance of discovering more new things.

We obviously already knew this in the world of startup enterprises. He’s telling us that in the world of medical research, it isn’t much different. Most startup enterprises fail. Most research is wrong. But it’s still worth trying.

You might just find the cure for cancer or find a way to be the next Facebook. But you might just as easily also be the next AltaVista. What else is there to do?