I study impact craters (those circle thingies on other planets, moons, asteroids, comets, etc.). A colleague recently pointed out a manuscript to me that demonstrated a new method to do something with craters. (I’m being purposely vague here to protect the situation.) It was an interesting manuscript, but it was submitted to an open archive (arxiv.org) where anyone can submit pretty much anything that seems sciencey. It has not been through the peer-review process.
Peer-review is not perfect. I’ve written about it before on this blog and discussed it on my podcast. But the purpose of peer-review is to weed out stuff that is obviously wrong. Things that may seem good to a general researcher, but to someone else who really knows the field, it clearly has issues. Other purposes of peer-review are to make sure the work is placed in proper context (usually by citing the reviewers’ works, but that’s a separate issue), making sure that the authors of the manuscript have explained themselves well, that their methods make sense, that they have explored alternative interpretations of their data, etc. In other words, do science “right.” Where “right” is in quotes because there is no formal set of rules by which one must play, but there are general guidelines and important pillars which people should uphold.
After it passes peer-review – if it passes peer-review – then it may be accepted by a journal and published. Some stuff that gets through peer-review is great. Some stuff is utter crap because the process isn’t perfect and because we don’t know everything, and the prevailing scientific opinion can shift with new information.
That is upended in today’s cut-throat world of journalism and a desire to be the first to publish about something that seems new and interesting.
I was contacted yesterday by a freelance reporter for the publication New Scientist. I’m not going to say the reporter’s name, but I have no qualms stating the publication. The reporter, coincidentally, wanted me to comment on the manuscript that had been submitted to arxiv.org. I refused. Here is what I wrote:
Thank you for writing. I am generally happy to comment about crater papers, and I would be happy to comment on this manuscript should it be accepted by the peer-review process. My concern at the moment is that the manuscript is only on an open server to which anyone can submit and it has not been vetted by researchers in the field beyond the authors themselves. The authors also used [specifics redacted] which have some significant omissions, and how that affects their results needs to be assessed by people who know all the ins and outs of their methods, which is not me.
I strongly recommend that you refrain from publishing about this work until it has made it through the peer-review process. It is easy to get excited about new techniques, but at the moment, it has not been vetted by other experts in the field, such that I think writing about it now is premature.
The reporter responded that I had a valid concern, he appreciated my advice, and he would discuss it with his editor.
Then just a few minutes ago, I heard from another friend in the field that she had been asked to comment for the story. She is taking a similar approach, which I greatly appreciate.
But this identifies, to me, a significant problem that those in both the scientific community and skeptic community have pointed out for years: Journalists don’t seem to care about vetting the science about which they write. Now, this could be an isolated example of an over-zealous reporter given the “OK” by their editor. Except it’s not. Too often we see articles about work just at the very edge of the field that offers great marvels and promises, only to hear nothing more from it because it was all based on extraordinarily preliminary efforts. Craters aren’t going to affect your daily life. But the issue here is a symptom of a greater problem. And I think that only if scientists and the reading public demand that reporters stop doing this will we see any sort of change.