I study impact craters (those circle thingies on other planets, moons, asteroids, comets, etc.). A colleague recently pointed out a manuscript to me that demonstrated a new method to do something with craters. (I’m being purposely vague here to protect the situation.) It was an interesting manuscript, but it was submitted to an open archive (arxiv.org) where anyone can submit pretty much anything that seems sciencey. It has not been through the peer-review process.
Peer-review is not perfect. I’ve written about it before on this blog and discussed it on my podcast. But the purpose of peer-review is to weed out stuff that is obviously wrong. Things that may seem good to a general researcher, but to someone else who really knows the field, it clearly has issues. Other purposes of peer-review are to make sure the work is placed in proper context (usually by citing the reviewers’ works, but that’s a separate issue), making sure that the authors of the manuscript have explained themselves well, that their methods make sense, that they have explored alternative interpretations of their data, etc. In other words, do science “right.” Where “right” is in quotes because there is no formal set of rules by which one must play, but there are general guidelines and important pillars which people should uphold.
After it passes peer-review – if it passes peer-review – then it may be accepted by a journal and published. Some stuff that gets through peer-review is great. Some stuff is utter crap because the process isn’t perfect and because we don’t know everything, and the prevailing scientific opinion can shift with new information.
That is upended in today’s cut-throat world of journalism and a desire to be the first to publish about something that seems new and interesting.
I was contacted yesterday by a freelance reporter for the publication New Scientist. I’m not going to say the reporter’s name, but I have no qualms stating the publication. The reporter, coincidentally, wanted me to comment on the manuscript that had been submitted to arxiv.org. I refused. Here is what I wrote:
Thank you for writing. I am generally happy to comment about crater papers, and I would be happy to comment on this manuscript should it be accepted by the peer-review process. My concern at the moment is that the manuscript is only on an open server to which anyone can submit and it has not been vetted by researchers in the field beyond the authors themselves. The authors also used [specifics redacted] which have some significant omissions, and how that affects their results needs to be assessed by people who know all the ins and outs of their methods, which is not me.
I strongly recommend that you refrain from publishing about this work until it has made it through the peer-review process. It is easy to get excited about new techniques, but at the moment, it has not been vetted by other experts in the field, such that I think writing about it now is premature.
The reporter responded that I had a valid concern, he appreciated my advice, and he would discuss it with his editor.
Then just a few minutes ago, I heard from another friend in the field that she had been asked to comment for the story. She is taking a similar approach, which I greatly appreciate.
But this identifies, to me, a significant problem that those in both the scientific community and skeptic community have pointed out for years: Journalists don’t seem to care about vetting the science about which they write. Now, this could be an isolated example of an over-zealous reporter given the “OK” by their editor. Except it’s not. Too often we see articles about work just at the very edge of the field that offers great marvels and promises, only to hear nothing more from it because it was all based on extraordinarily preliminary efforts. Craters aren’t going to affect your daily life. But the issue here is a symptom of a greater problem. And I think that only if scientists and the reading public demand that reporters stop doing this will we see any sort of change.
Exposing PseudoAstronomy 
Reblogged this on Talmidimblogging.
Comment by Vincent S Artale Jr — March 9, 2018 @ 2:02 pm |
ArXiv is not merely “an open archive” – it is the depository for preprints in many science fields and often considered a model for the future of scientific publishing with open peer review (they are currently restructuring the system to facilitate that aspect even more). Some stuff there has gone through classical peer review (and some has even been published; putting it then also on ArXiv can double the number of citations a paper gets) while some has just been submitted – but total fringe stuff is weeded out. Papers on ArXiv are fair game for science journalism and they absolutely should be: it’s here where science happens and can be watched in progress as hardly anywhere else. And being a witness of the process is key to successful and productive science communication. Scientists refusing to comment on ArXiv papers when asked for advice by the public and media in particular are making a grave mistake.
Comment by skyweek — March 9, 2018 @ 5:53 pm |
I completely disagree with you. This paper has been submitted to a journal. It’s on arXiv without any peer-review at the moment. I’m not going to comment on an article that is in the peer-review stages and has not finished, especially when it’s something completely new to the field. Papers that are on arXiv that have already gone through peer-review are perfectly fine. It’s a free place not behind a paywall. But the idea that anyone can submit anything there and then it should be talked about in the public as valid is one that I will not get behind.
Comment by Stuart Robbins — March 9, 2018 @ 6:07 pm |
Should the paper in question be https://arxiv.org/abs/1803.02192 (no other “crater” papers on ArXiv in the past week) then hail to the science writer for spotting it: this one is newsworthy indeed, for applying a well-known technique to a new astronomical problem – apparently with success. And it is also a perfect case for open peer review, even crowd sourcing as its relevance rests on the algorithm – which others can and should try out on further test and real cases. The official reviewer for Icarus may not have the skills – or, usually, time – to do that with the needed effort, but some of the readers of an article about the work might well be in a position to do it. Welcome to open science in the 21st century!
Comment by skyweek — March 9, 2018 @ 6:33 pm
I will neither confirm nor deny that being the manuscript in question.
You and I clearly seem to have different opinions about this. My opinion comes from seeing too many science-by-press-release events only to never have any follow-up when it turns out to be a dud. The general public just sees the marvels touted by the initial story and nothing else. Additionally, you may be contradicting yourself, for you state this “is also a perfect case for open peer review,” and I am advocating for peer review before a manuscript is part of a mainstream story.
I will also state without fear of giving away what manuscript this was, that the subject under-which the reporter asked for my comment was a significant over-statement of the manuscript’s work and ignored other work that had done similar things. Hence why I pointed out in my post that one part of peer review is to ensure that the work is placed within proper context.
Comment by Stuart Robbins — March 9, 2018 @ 7:12 pm
We are on very much the same page when it comes to science -by-press-release: all too often the latter – which is an advertisement for a paper and even more for the institution(s) it came from – gets just copy-pasted into a story. Happens so frequently that there is even a word for that abomination: churnalism. But press releases are usually issued about papers that have been through peer review or even published already – so this is the exact opposite to “my world” in which educated science journalists find and vet breaking news, be it on social media (yes, serious science debate can be found there), in conference talks or on ArXiv. O.k., sometimes you stumble (but so do the churnalists when the paper was bad in spite of all reviewing), but overall it’s so much more rewarding … in some cases even forcing you to do your own math.
Comment by skyweek — March 9, 2018 @ 7:42 pm |
Indeed they are, looking forward to episode 170, whatever it turns out to be.
Comment by Graham — June 14, 2018 @ 8:15 am |
Sadly, modern journalism is all about the “clicks/eyeballs/whatever”, and anything that gets in the way of that, like the scientific method, is irrelevant. The journalists figure that a couple of weeks after publication, almost everybody will have forgotten about it, so who cares if it’s accurate? The penalties nowadays for writing a misleading or incorrect article are (at most) a slap on the wrist.
Besides, most news articles now read more like opinion pieces, with objective reporting looked down on as antiquated or quaint. One has to really dig to find out what’s really going on, due to the hyperbole and padding in almost piece considered newsworthy.
By the way, I’m also waiting for episode 170. Take your time with it, because I’d rather it be well-made and not rushed.
Comment by Rick K. — June 14, 2018 @ 6:17 pm |
“…in almost every piece considered newsworthy.” Sorry about that.
Comment by Rick K. — June 14, 2018 @ 6:19 pm |