The Replication Crisis

Science is facing a “reproducibility crisis” where more than two-thirds of researchers have tried and failed to reproduce another scientist’s experiments, research suggests.

This is what the BBC have reported. But how true is it? Is the world of science having a crisis when it comes to repeating experiments? And if so, what are the consequences?

Reproducibility is part of the foundation on which science is built. Science is our best method available to us in determining how nature works, and if our hypotheses are to have any basis in the real world, then the observations on which those hypotheses are based must be reproducible. To the average person, this doesn’t sound like a issue. After an interesting scientific paper is published, can’t other scientists just do what the scientists publishing the paper did? Aren’t the methods section there for a reason? However, as any biologist well knows, it’s nowhere near that simple. Many living systems simply don’t behave the same way all the time, they have minds of their own and do their own thing. It doesnt matter that you don’t want them to. Arguably the biggest issue however, is the lack of reward for reproducing the work of others. A scientist is not going to get a grant to reproduce the results of others, it’s already been done once, what’s the need to do it twice?

Despite being a defining feature of science, reproducibility is more an assumption than a practice in the present scientific ecosystem. Incentives for scientific achievement prioritize innovation over replication. – Reproducibility Project: Cancer Biology

Scientists go into science to make new discoveries, to make their own contribution to their field, not to repeat work already done by their peers.

An article published in Nature last May, reveals just how bad this problem has gotten. Of the 1576 people asked, more than 70% have tried and failed to replicate the work of others, with 52% of respondents agreeing that there is a reproducibility crisis (Figure 1).reproducibility-graphic-online1

Figure 1 – The results from the Nature survey. A significant amount agreed that there is at least a slight crisis when it comes to repeating experiments. Source.

Though the vast majority of respondents had failed to repeat an experiment (Figure 2), few said they had been contacted by others who had been unable to reproduce their work. This highlights an issue within the scientific community where if experimenters reach out to the original authors, they might appear as accusing them of publishing bad data. This means that scientists aren’t sharing vital information which could lead to higher repeatability of the work. Furthermore, a minority reported they had tried to publish a replication study, with only 37% of those saying they had been able to.

reproducibility-graphic-online3

Figure 2 – The ability to repeat experiments is not universal, with some disciplines being better than others. Here the graph shows the percentage of those asked in each field, who had failed to repeat an experiment. Even more shockingly is the second bar, showing just how many people are unable to even repeat their own work. Source.

Those who took the survey said the cause of this crisis was mostly due to two factors – pressure to publish, and selective reporting of data. When you are under pressure to publish new findings before anyone else, often you don’t repeat your own experiments enough, or you only pick the data that shows what you need it to. The data that doesnt fit with the idea just gets discarded which means that others will get different results when they repeat. In addition, Judith Kimble of the University of Wisconsin-Madison suggested that bureaucracy is having a larger impact, where less and less time is being dedicated to doing the research, as one competes for grants in order to do new research – something that seems to conflict wouldn’t you think?

All is not lost however, there are ways in which scientists can improve the repeatability of their work. The results from the survey show that nearly 90% of the respondents believe that “More robust experimental design” and “better mentorship”are the keys to doing so. These two approaches were even ranked higher than options providing incentives for reproducibility. So though there is an issue with being able to repeat work done in science, the ways in which to fix this have been thought about and indeed begun to be implemented. Approximately one-third respondents said that the labs they worked in had taken measures to improve the reproducibility over the past five years, though it varied across disciplines. The most common effort was to ask someone else within a lab to repeat the work.

If we don’t act now, however, the moment will be lost and science may begin to crumble as part of its foundations falls from under it.

Further Reading:

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s