Introduction
Many people outside of mainstream science – such a conspiracy theorists, psi researchers, UFOlogists, and others – seem to have a beef with the process of peer-review. And, some mainstream scientists do, too. The purpose of this post is really to address why we have peer-review, why it’s important, why science really does need it in order to meet its goals, and to be fair, address some of its weaknesses.
Why Am I Addressing This
It really never registered much to me that fringe researchers would knock the peer-review system. It kinda went in one ear and out the other. Then, I was listening to the December 22, 2008, episode of the podcast “Skeptiko” with Alex Tsakiris where he spends several minutes complaining about how mainstream scientists “do” science. One of his big complaints and something that he called “stupid” (that’s a quote) was the embargo on releasing early results. He thinks that results should be released as they come in.
I made the following observations on an online forum:
Alex really seems to have no grasp of how science is actually done. At about 20 minutes into his last podcast, he states, “I want to break the traditional science rule about not talking about results until they’re published because, well, first of all, I think it’s a stupid rule [and he laughs] …” Results usually aren’t announced early for several reasons, not the least of which is that it hasn’t passed any peer review yet.
For example, I could do some ground-breaking research for a year and get this great result and then talk about it, or I could pass it by peers first only to have them discover that I’ve not accounted for some small factor that will dramatically reduce the significance.
Another reason is that preliminary results are just that – preliminary. One of my research projects at the moment is to generate a complete global database of Mars craters to ~1.5 km in diameter. I’ve done that now for about 30% of the planet. I could go ahead and release results and get more papers out of it, or I could wait until the whole thing is finished and I have all the statistics in place to back up my conclusions. This is especially necessary because Mars is not all the same, and craters from different regions have different properties, so me releasing early results that make broad conclusions could easily turn out to be fallacious once the entire project is done.
And as usual, Alex just seems to not get it. His results are going the way he thinks they should, so he’s releasing them early and claiming at least a cautious victory “so far.” This is also partly why I’m not a giant fan of his “open source science” — you really DO need training in science before you can do it “properly” — learning to take into account all these things that you may not otherwise, normally, think of.
That was really about releasing results early, and a little about peer-review.
Then, I was listening to the January 15, 2009, episode of Coast-to-Coast AM with Richard Hoagland. Among other things, he made the following statement: “You follow your curiosity, which is what science is supposed to be. It’s not supposed to be a club or a union or a pressure group that doesn’t want to get too far out of the box ’cause of what the other guys will think about you. … This concept of ‘peer review’ … is the thing which is killing science.”
It was with that line that I decided I should write this post.
Why We Have Peer-Review
Peer-review is important. The whole point of peer-review is so that your findings – your data and conclusions – are subjected to the review of your peers.
To use a reductio ad absurdum logic, if we didn’t have this process, then what anyone says is basically dogma with no chance of rebuttal. For example, if there weren’t a process of peer-review, I can say 2+2=1. You may say 2+2=5. And someone else may say 2+2=4. How would anyone know which is correct? The obvious answer in this contrived example is that everyone knows that 2+2=4. But how? Because you ask someone else, and they tell you? That’s peer-review.
In science, the purpose of peer-review is really just that. Your peers (other people who study what you study) look at your findings and make sure that in their opinion, you have followed the proper data gathering methods (so you took 2 apples and 2 oranges and laid them down as opposed to meditated and asked your spirit guide) and you reached the conclusions that are appropriate for the data you gathered (you then count all the pieces of fruit and come up with 4 instead of your spirit guide saying that 2+2 is really 7).
The purpose of peer-review is really nothing more than that, and it is nothing less than that.
Why Science Needs Peer-Review
It is often said that science is “self-correcting” over time. What this means is that if science has led to erroneous conclusions that did pass through the peers at the time, that ultimately the errors will be worked out because the process and data-collecting are repeated over and over again by others. A good example of this is gravity. Newton developed his Theory of Gravity. It was used for centuries. Repeated experiments showed it to be accurate.
But, some of them didn’t. Some showed slight deviations (like Mercury’s orbit). Then, another researcher came along (Einstein) and showed that Newton’s theory needed to be modified in order to account for ginormous masses and accelerations. Without the process of people reviewing predictions and measurements relevant to gravity, then we would not know that Newton didn’t have the whole picture. And even today, a century later, people are still testing Einstein’s theories, making more and more measurements to test them, subjecting them to the process of peer-review.
Hoagland’s Claims
I am not saying that these are representative of the general fringe community’s problems with peer-review, simply that they are what I have observed to be the general complaint. It’s fairly well-said by Richard Hoagland, this quote continuing from the one I ended with above:
“It’s not the peer review so much as the invisible, anonymous, peer-review. Basically, before a paper can get published, … you know you have to go through so many hurdles, and there’s so many chances for guys who have it ‘in for you,’ who don’t like you, or who don’t like the idea you’re trying to propose in a scientific publication, can basically … stick you in the back … and you never know!
“One of the tenants of the US Constitution … is that you have the right to confront your accuser. In the peer-review system, which has now been set up for science, … the scientist – which [sic] is basically on trial for an idea – because that’s what it is, by any other name it’s really a trial, is-is attacked by invisible accusers called ‘referees,’ who get a chance to shaft the idea, kill the idea, nix the paper, tell the editor of whatever journal, ‘Oh, this guy’s a total wacko …’ and you never have the opportunity to confront your accuser and demand that he be specific as to what he or she has found wrong with your idea.”
My Response to Hoagland
I don’t know what journals he’s talking about, but for all the ones I know of, his claims are wrong. Just as with the US court system, you have appeals in journals. If the first reviewer does not think your paper should get in, then you can ask the editor to get another opinion. You’re never sunk just because one reviewer doesn’t like you and/or your ideas.
As to the anonymity, while I personally don’t like it, it’s necessary. Without a referee having the ability to remain anonymous, they cannot always offer a candid opinion. They may be afraid of reprisals if they find errors (after all, grants are also awarded by peer-review). They may also not want to hurt someone’s feelings (as teenagers today are finding, it’s much easier to break up via Facebook or a txt message than in person — it’s the same with anonymity in peer-review). They may have their own work on the subject they think you should cite but don’t want to appear narcissistic in recommending it. In short, there are many very good reasons to remain anonymous to the author(s).
However, they are not anonymous to the editor or the editorial staff. If there are problems with a reviewer consistently shooting down ideas that they have an otherwise vested interest in, then the editors will see that and they will remove the reviewer.
I also want to point out something my officemate is fond of saying: “Science is not a democracy, it’s a meritocracy.” Not every idea deserves equal footing. If I come up with a new idea that explains the universe as being created by a giant potato with its all-seeing eyes (Dinosaurs fans, anyone?) then my new idea that I just made up should not deserve equal footing with the ones that are backed up by centuries of separate, independent evidence. The latter has earned its place, the former has not.
That is something that most fringe researchers seem to fail to grasp: Until they have indisputable evidence for their own ideas that cannot be otherwise easily explained by the current paradigm, then they should not necessarily be granted equal footing. Hoagland’s pareidollia of faces on Mars does not deserve an equal place next to descriptions of the martian atmosphere.
The Cons of Peer-Review
There are bad points to peer-review, though they really are only when there is an abuse of it. There is a faculty member back in my undergraduate institution who likes to tell the story of a young astronomer who submitted a paper about the value of the Hubble Constant (a measure of how rapidly the universe is expanding). The paper was sent to a reviewer who had his own ideas, and the young astronomer’s were not the same as his. So, he sat on the paper. He wrote a rebuttal to it. And he had the rebuttal published before he got to her paper.
That is an abuse of the system. I think that every scientist would admit that, and we strive to not be “that person.” After all, “that person” is now fairly blacklisted from polite astronomy society, and, as I’ve just done, people talk behind his back about him and how crummy he was.
In the vast sea of peer-review, however, there are just a few drops of “those people.” Most reviewer comments are helpful. They usually think of things you didn’t, and they only serve to make your results stronger.
Final Thoughts
The process of peer-review in science is an old one and one that is important to the essence of what science is and what it is supposed to do. If someone continuously complains about it, then the first thing you should do is to ask yourself what the motivation may be behind their ideas. Is it because they happened to get burned by one reviewer? Or is it perhaps because their ideas really don’t pass any scientific muster, they don’t fit with every other observation, and they require an extraordinary new premise to be true without sufficient evidence to back it up?
Exposing PseudoAstronomy 