Two basic scientific principles, transparency and reproducibility, which reportedly “saved the universe as we know it,” could also help bring the scientific method back to chemical risk assessment. This post will attempt to explain why these principles are critical and how the scientific community could help make that happen.
Back in 2011, scientists appeared to make a startling discovery while working at the Large Hadron Collider at the multi-billion-dollar CERN laboratory outside Geneva; they found a particle that could travel faster than the speed of light. However, as Einstein’s theory of special relativity makes clear, that kind of observation shouldn’t be possible. Nothing can travel faster than light, right?
But, wait a minute. Weren’t the people who made this unlikely discovery supposed to be the “Einsteins” of our generation? And doesn’t “new science” almost always trump the old? And by the time the study makes its way into the daily news, hasn’t the data been checked and rechecked, the experiment replicated, and the findings confirmed?
Well, no, not really.
In this exciting case of faster-than-light particle travel, the world seemed to willfully suspend disbelief – if only for a little while. That still gave the media plenty of time to put together a what-if story and share it with the world. And why not?
But soon after the media firestorm came the proverbial wet blanket. Researchers reviewed the data and ran the experiment again, and again, and again before they made a very different kind of discovery. The original experiment that found faster-than-light travel and that would have upended a century of scientific research was the result of… a loose cable.
Of all the billion and one pieces that went into this multi-billion-dollar investment beneath the Swiss and French Alps, something as simple as a loose cable was ultimately responsible for the apparent threat to Einstein’s legacy, not to mention a lot of newspaper ink.
So what are we supposed to take away from what happened in 2011?
The video above, produced by the National Science Foundation, does a pretty good job of explaining why researchers can jump the gun and forget about the importance of reproducibility in science.
The story has a happy ending and productive conclusion. Thanks to the willingness on the part of the CERN researchers to share their data, methods, and allow other scientists the opportunity to run the test themselves, sound science ultimately prevailed.
Transparency and reproducibility: Two hallmarks of the scientific method
The thing about scientific data is that it isn’t meant to be mute – it’s supposed to be presented so it can speak for itself. That is, the researcher conducting the experiment shouldn’t need to be there explaining away discrepancies or complicated interpretations. However, without the data, other scientists cannot possibly try to reproduce the results and verify them.
This video offers a hilarious but real illustration of just how hard it can be to get the original data from an important study:
It doesn’t have to be this way. The Center for Open Science (COS) in Charlottesville, VA is spearheading an effort to restore greater transparency and reproducibility in scientific research, which the COS says are “critical aspects of science that are frequently overlooked in the pursuit of novelty and impact.” So far, the COS has the support of at least 41 organizations and 114 journals – but there are a few key players still missing, and that has to change.
Brian Nosek, Executive Director of the Center for Open Science and Professor of Psychology and the University of Virginia, and Chris Chambers, Professor of Cognitive Neuroscience at Cardiff University, make it clear why the entire scientific community needs to rally together:
Transparency and reproducibility are the beating heart of the scientific enterprise. Transparency ensures that all aspects of scientific methods and results are available for critique, compliment, or reuse. This not only meets a social imperative, it also allows others to test new questions with existing data, makes it easier to identify and correct errors, and helps unmask academic fraud. Transparent practices such as sharing data and computer code, in turn, safeguard reproducibility: the idea that for a scientific observation to count as a discovery it must reveal something real and repeatable about the natural world…. As any scientist will tell you, the fierce competition of academia rewards those who secure large grants and publish innovative – if tentative – findings in prestigious journals, not scholars who instead focus on being transparent and careful. The incentives that drive individual scientists are out of step with what is best for science as a social enterprise.
As ACC’s Nancy Beck noted in our blog last November, more than 30 major scientific journals, have made a new commitment to improving reproducibility, rigor, transparency and independent verification. This is great progress; however, not all the journals that toxicologists and risk assessors publish in have signed on to this effort.
A Washington Post article published earlier this year by Joel Achenbach, noted that “too often, experimental results can’t be reproduced” and that “an irreproducible result is inherently squishy.” As Achenbach reports:
And so there’s a movement afoot, and building momentum rapidly. Roughly four centuries after the invention of the scientific method, the leaders of the scientific community are recalibrating their requirements, pushing for the sharing of data and greater experimental transparency.
It’s time we make all the pieces visible and publically available so the broader community can understand how they fit together, this includes the opportune to attempt to take apart and reassemble the data to see if we get the same results.
The new Transparency and Openness Promotion (TOP) guidelines, developed by COS, lay out eight standards for journals that nudge the scientific community to provide greater openness. Many important scientific journals in the fields of toxicology, environmental health, and risk assessment currently do not meet the standard. They are at level 0. What can the scientific community do to improve the approaches of researchers and publishers, including important and impactful journals such as Toxicological Sciences and Environmental Health Perspectives (EHP), to ensure movement towards achieving a higher TOP standard?
TOP Guidelines courtesy of Open Science Framework
Perhaps a good place to start building a more transparent and open research community would be to have researchers and publishers agree that we should all strive to meet at least the level 1 goals for published articles. For that that to happen, influential journals need to increase requirements for potential authors to encourage more transparency regarding the information and methods used to support conclusions in publications.
These basic but important changes would help foster the public’s trust and confidence in scientific research and would thus increase the value and credibility of published studies. This would be of benefit to journals, authors and the public as a whole.
