Exposing PseudoAstronomy

January 22, 2009

The Purpose of Peer-Review in Science


Introduction

Many people outside of mainstream science – such a conspiracy theorists, psi researchers, UFOlogists, and others – seem to have a beef with the process of peer-review. And, some mainstream scientists do, too. The purpose of this post is really to address why we have peer-review, why it’s important, why science really does need it in order to meet its goals, and to be fair, address some of its weaknesses.

Why Am I Addressing This

It really never registered much to me that fringe researchers would knock the peer-review system. It kinda went in one ear and out the other. Then, I was listening to the December 22, 2008, episode of the podcast “Skeptiko” with Alex Tsakiris where he spends several minutes complaining about how mainstream scientists “do” science. One of his big complaints and something that he called “stupid” (that’s a quote) was the embargo on releasing early results. He thinks that results should be released as they come in.

I made the following observations on an online forum:

Alex really seems to have no grasp of how science is actually done. At about 20 minutes into his last podcast, he states, “I want to break the traditional science rule about not talking about results until they’re published because, well, first of all, I think it’s a stupid rule [and he laughs] …” Results usually aren’t announced early for several reasons, not the least of which is that it hasn’t passed any peer review yet.

For example, I could do some ground-breaking research for a year and get this great result and then talk about it, or I could pass it by peers first only to have them discover that I’ve not accounted for some small factor that will dramatically reduce the significance.

Another reason is that preliminary results are just that – preliminary. One of my research projects at the moment is to generate a complete global database of Mars craters to ~1.5 km in diameter. I’ve done that now for about 30% of the planet. I could go ahead and release results and get more papers out of it, or I could wait until the whole thing is finished and I have all the statistics in place to back up my conclusions. This is especially necessary because Mars is not all the same, and craters from different regions have different properties, so me releasing early results that make broad conclusions could easily turn out to be fallacious once the entire project is done.

And as usual, Alex just seems to not get it. His results are going the way he thinks they should, so he’s releasing them early and claiming at least a cautious victory “so far.” This is also partly why I’m not a giant fan of his “open source science” — you really DO need training in science before you can do it “properly” — learning to take into account all these things that you may not otherwise, normally, think of.

That was really about releasing results early, and a little about peer-review.

Then, I was listening to the January 15, 2009, episode of Coast-to-Coast AM with Richard Hoagland. Among other things, he made the following statement: “You follow your curiosity, which is what science is supposed to be. It’s not supposed to be a club or a union or a pressure group that doesn’t want to get too far out of the box ’cause of what the other guys will think about you. … This concept of ‘peer review’ … is the thing which is killing science.”

It was with that line that I decided I should write this post.

Why We Have Peer-Review

Peer-review is important. The whole point of peer-review is so that your findings – your data and conclusions – are subjected to the review of your peers.

To use a reductio ad absurdum logic, if we didn’t have this process, then what anyone says is basically dogma with no chance of rebuttal. For example, if there weren’t a process of peer-review, I can say 2+2=1. You may say 2+2=5. And someone else may say 2+2=4. How would anyone know which is correct? The obvious answer in this contrived example is that everyone knows that 2+2=4. But how? Because you ask someone else, and they tell you? That’s peer-review.

In science, the purpose of peer-review is really just that. Your peers (other people who study what you study) look at your findings and make sure that in their opinion, you have followed the proper data gathering methods (so you took 2 apples and 2 oranges and laid them down as opposed to meditated and asked your spirit guide) and you reached the conclusions that are appropriate for the data you gathered (you then count all the pieces of fruit and come up with 4 instead of your spirit guide saying that 2+2 is really 7).

The purpose of peer-review is really nothing more than that, and it is nothing less than that.

Why Science Needs Peer-Review

It is often said that science is “self-correcting” over time. What this means is that if science has led to erroneous conclusions that did pass through the peers at the time, that ultimately the errors will be worked out because the process and data-collecting are repeated over and over again by others. A good example of this is gravity. Newton developed his Theory of Gravity. It was used for centuries. Repeated experiments showed it to be accurate.

But, some of them didn’t. Some showed slight deviations (like Mercury’s orbit). Then, another researcher came along (Einstein) and showed that Newton’s theory needed to be modified in order to account for ginormous masses and accelerations. Without the process of people reviewing predictions and measurements relevant to gravity, then we would not know that Newton didn’t have the whole picture. And even today, a century later, people are still testing Einstein’s theories, making more and more measurements to test them, subjecting them to the process of peer-review.

Hoagland’s Claims

I am not saying that these are representative of the general fringe community’s problems with peer-review, simply that they are what I have observed to be the general complaint. It’s fairly well-said by Richard Hoagland, this quote continuing from the one I ended with above:

“It’s not the peer review so much as the invisible, anonymous, peer-review. Basically, before a paper can get published, … you know you have to go through so many hurdles, and there’s so many chances for guys who have it ‘in for you,’ who don’t like you, or who don’t like the idea you’re trying to propose in a scientific publication, can basically … stick you in the back … and you never know!

“One of the tenants of the US Constitution … is that you have the right to confront your accuser. In the peer-review system, which has now been set up for science, … the scientist – which [sic] is basically on trial for an idea – because that’s what it is, by any other name it’s really a trial, is-is attacked by invisible accusers called ‘referees,’ who get a chance to shaft the idea, kill the idea, nix the paper, tell the editor of whatever journal, ‘Oh, this guy’s a total wacko …’ and you never have the opportunity to confront your accuser and demand that he be specific as to what he or she has found wrong with your idea.”

My Response to Hoagland

I don’t know what journals he’s talking about, but for all the ones I know of, his claims are wrong. Just as with the US court system, you have appeals in journals. If the first reviewer does not think your paper should get in, then you can ask the editor to get another opinion. You’re never sunk just because one reviewer doesn’t like you and/or your ideas.

As to the anonymity, while I personally don’t like it, it’s necessary. Without a referee having the ability to remain anonymous, they cannot always offer a candid opinion. They may be afraid of reprisals if they find errors (after all, grants are also awarded by peer-review). They may also not want to hurt someone’s feelings (as teenagers today are finding, it’s much easier to break up via Facebook or a txt message than in person — it’s the same with anonymity in peer-review). They may have their own work on the subject they think you should cite but don’t want to appear narcissistic in recommending it. In short, there are many very good reasons to remain anonymous to the author(s).

However, they are not anonymous to the editor or the editorial staff. If there are problems with a reviewer consistently shooting down ideas that they have an otherwise vested interest in, then the editors will see that and they will remove the reviewer.

I also want to point out something my officemate is fond of saying: “Science is not a democracy, it’s a meritocracy.” Not every idea deserves equal footing. If I come up with a new idea that explains the universe as being created by a giant potato with its all-seeing eyes (Dinosaurs fans, anyone?) then my new idea that I just made up should not deserve equal footing with the ones that are backed up by centuries of separate, independent evidence. The latter has earned its place, the former has not.

That is something that most fringe researchers seem to fail to grasp: Until they have indisputable evidence for their own ideas that cannot be otherwise easily explained by the current paradigm, then they should not necessarily be granted equal footing. Hoagland’s pareidollia of faces on Mars does not deserve an equal place next to descriptions of the martian atmosphere.

The Cons of Peer-Review

There are bad points to peer-review, though they really are only when there is an abuse of it. There is a faculty member back in my undergraduate institution who likes to tell the story of a young astronomer who submitted a paper about the value of the Hubble Constant (a measure of how rapidly the universe is expanding). The paper was sent to a reviewer who had his own ideas, and the young astronomer’s were not the same as his. So, he sat on the paper. He wrote a rebuttal to it. And he had the rebuttal published before he got to her paper.

That is an abuse of the system. I think that every scientist would admit that, and we strive to not be “that person.” After all, “that person” is now fairly blacklisted from polite astronomy society, and, as I’ve just done, people talk behind his back about him and how crummy he was.

In the vast sea of peer-review, however, there are just a few drops of “those people.” Most reviewer comments are helpful. They usually think of things you didn’t, and they only serve to make your results stronger.

Final Thoughts

The process of peer-review in science is an old one and one that is important to the essence of what science is and what it is supposed to do. If someone continuously complains about it, then the first thing you should do is to ask yourself what the motivation may be behind their ideas. Is it because they happened to get burned by one reviewer? Or is it perhaps because their ideas really don’t pass any scientific muster, they don’t fit with every other observation, and they require an extraordinary new premise to be true without sufficient evidence to back it up?

7 Comments »

  1. Peer review is a flawed process and abuses occur. But why should this surprise anyone? It’s carried out by humans.

    Despite its flaws,shortcomings, unfairness, destructiveness, etc, etc, we haven’t come up with something better. I encourage people to keep trying but until we do (come up with something better), we’re stuck with an imperfect process that works well enough most of the time for it to support scientific progress, the development of new ideas and careers and the careful vetting of scientific data.

    To want to disregard a process that does a tremendous amount of good because harm sometimes is done is to doom us to doing nothing. There is a place for open source, non-peer reviewed science but it will remain a fringe element in the scientific literature.

    There is real value in multiple reviews by independent experts and editors carrying out due diligence. Where the abuses occur they should be caught, the abusers “outed” and punished. While I agree that this doesn’t happen most of the time, that is one of the consequences of living in a political world.

    Comment by teachdna — January 22, 2009 @ 7:02 pm | Reply

  2. Things are changing rapidly with digital media and networks. The peer review process will change with that as well. But it’s important to remember (echoing what teachdna said) that the solution is to improve the peer review we have, not to turn it into something different, for example something like a blogospheric free-for-all where anyone can comment and no expertise is necessary. That has its place but not in professional scientific publication.

    Comment by Ganz — January 22, 2009 @ 9:08 pm | Reply

  3. I’m continually surprised not by how badly peer review works, but by how well it works. A couple of years ago, I submitted a *long* paper covering about 3 years of lab work, and I thought all the i’s were dotted and the t’s were crossed. One reviewer basically said, “This is big, and the work was really hard to do. Publish it.” The other was very thorough and nit-picky, and found two somewhat important mistakes. I was REALLY GLAD to have a chance to fix the problems before the paper was published. By being critical, the reviewer improved the quality of the paper and helped my career.

    As a referee, I’ve struggled with how critical I should be of other scientists work — particularly when I’m reviewing a paper that was written by people who are more established in the field than me, or people who I know personally. Last year I read a paper by a grad student that had some serious problems. I really took the author to task in my review. I felt so guilty I lost sleep over it. But when the paper came back, it was vastly improved. Math and physics errors corrected, prose cleaned up, logic tightened. After I saw the revisions, I felt much better. Maybe I also improved the quality of someone’s paper, and helped their career.

    There are certainly individuals who abuse the system, but I think most of us are struggling to provide truly constructive criticism — and devoting a lot of time to the effort. Anonymity is an important part of the process, particularly in small fields where most people are on a first name basis. I couldn’t have provide the harsh review to the grad student if he had know it was me. I assume the anonymous reviewer who improved my lab paper felt similarly.

    Comment by Hanna — January 23, 2009 @ 10:30 am | Reply

  4. […] have posted once before about Skeptiko podcast host Alex Tsakiris in my post about The Importance of Peer-Review in Science. The purpose of that post was to primarily show that peer review is an important part of the […]

    Pingback by Skeptiko Host Alex Tsakiris: On the Non-Scientifically Trained Trying to Do/Understand Science « Exposing PseudoAstronomy — May 29, 2010 @ 1:06 pm | Reply

  5. […] probably recognize it from the two previous posts I’ve written about him on this blog. The first was on the purpose of peer-review in science because Alex (among others) were talking about how peer-review was a flawed process and also that […]

    Pingback by Skeptiko Host Alex Tsakiris on Monster Talk / Skepticality, and More on How to Spot Pseudoscience « Exposing PseudoAstronomy — December 9, 2011 @ 11:58 am | Reply

  6. I am sorry I need to point this out, but I see this mistake made constantly by so many people: The past tense of ‘lead’ is ‘led.’ I know you are a smart guy, so I’m assuming it was a typo. Great ‘casts!

    Comment by joe — March 8, 2012 @ 8:06 am | Reply

    • I got Bs in English.

      Comment by Stuart Robbins — March 8, 2012 @ 10:13 am | Reply


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: