That system is the peer review of scientific papers. You might be inclined to dismiss peer review – a complex system that varies and sometimes falters in practice – as of little interest to those outside of scientific research and publishing. But this time last year I embarked on a series of conversations with those at the frontline of communicating with the public on science and medical matters and I discovered a huge appetite for guidance to help the public distinguish between dubious research claims and good science, and that knowledge about the peer review system helps to sate that hunger.
From Al foil causing Alzheimer's to the potential of magnetic bandages to help wounds heal faster, we are inundated on a daily basis with quasi-scientific claims. For those of us lucky enough to be on the ‘inside’ of the science scene, we can normally sort the science from the pseudo-science, or know someone else who can help us do this. But how can those on the ‘outside’ weed out the impostors and check the veracity of the claims they encounter? In order to make informed decisions about government policy, or our family's health, we all need to know what research claims to believe and how to recognize them.
At the end of last year, following workshops with charity helpline staff, medical workers, members of the UK parliament, and high school teachers among others, we developed a guide, “I don't know what to believe…”, which explains how the peer review system works and why. As scientists know, peer review is an essential dividing line for judging what is scientific and what it speculation and opinion – it is an arbiter of scientific quality. Few other professions have an in-built quality checking process before data is made public. Peer review is not merely about ticking boxes and rubber stamping a paper, but is about the critical engagement with someone else's work
“I don't know what to believe…” explains that work that has been peer-reviewed has been judged valid, significant, and original by other scientists working in the field. It also tells the public how to dig a little deeper for evidence that scientific findings are published in a peer-reviewed journal and what that evidence might look like. In essence, as someone commented to me during the workshops, knowing if research has been peer reviewed makes it possible for the public to know what is mere opinion and what is underpinned by good research.
This is all well and good, but peer review has been around since the 18th century and remains relatively unknown outside the scientific community, raising the question: why don't scientists promote peer review more? Rather than celebrate the system scientists instead appear to concentrate on its flaws, highlighting when it goes wrong, like when a fraudulent paper slips the net, or when it is hard to get a new piece of research published.
Unfortunately, it is often the case that the negative examples draw attention to the system. Scientific fraud has been the subject of much media scrutiny of late, with stem-cell scientist Hwang Woo-Suk and his fabricated results sparking a range of exposés about how scientists work. In our eagerness to discuss the pitfalls, we forget that such examples of serious fraud are rare. Indeed, by looking at the negative (that peer review can't work as a fraud detection system) we miss out on the positive (without the publication of Woo-Suk's work we may not have known for years that the work was unsound). As it was, the fraud was exposed within six months.
I am not suggesting that peer review is perfect, it isn't. Like any large-scale system involving many thousands of people and different organizations around the world, it can go wrong. But unpublished research is far worse. Scientists can't use it to repeat research and develop work as it would be impossible to know what to take seriously and what to discard. If scientists aren't able to function like this, what hope would the public have? As a society we can't base decisions on data that has a high chance of being flawed.
That scientists have gripes with the system, which was recently described to me as “what makes science scientific” is understandable; peer review is core to scientific practices and scientists have an interest in making it work better. However, we need to make a distinction between the technical discussion of how peer review works for scientists and what the public needs.
The reaction to “I don't know what to believe…” has been overwhelming with over 50?000 paper copies requested and more than 20?000 copies downloaded since its launch in November. The message is simple: when faced with new research claims we should all ask, has it been published in a peer-reviewed journal? If not, why not? Knowing how science publishing works can help people to judge the research claims they are confronted with daily; to sort the claims that have been exposed, scrutinized, and criticized by other scientists in the field from those that have not been subjected to such a rigorous process.
Further reading
[1] Sense About Science's short guide to peer review,“I don't know what to believe…” is available online atwww.senseaboutscience.org/peerreview. Hard copies (free of charge) can be ordered through the website or by e-mailing publications@senseaboutscience.org