A Post About Not Posts – NASA Grant Review Panels and Confidentiality

Introduction

I was recently asked to serve on a NASA grant review panel. What that means is that I’m expected to independently read several scientific proposals (ideally within my area or remotely within my area of expertise), write up what I think of them and whether I think they should be funded, and then fly to Washington, D.C., to meet with other people and as a group, over the course of a week, make a recommendation to the program officer about what should be funded.

I was planning to do a series of blog posts talking about this process — the details of how it works, as I was going through it. I thought it would be helpful to everyone to understand how it works — both the general public and young scientists. The plan was to anonymize everything.

When I ran this by the program manager, I was told that the desire for confidentiality was paramount, and he was not comfortable with me blogging about it. Hence, I’m making this post, before the panel meets, to avoid any tainting by that, and to describe why I won’t be blogging about it.

Confidentiality

I can immediately hear conspiracy theorists screaming that I’m a NASA stooge, that NASA wants to keep everything secret, that it’s public money so taxpayers should be able to know every detail of the process, etc. Here’s why that’s not true, and why everything needs to be kept confidential. You’ll notice that I haven’t even said what grant review panel I’m on.

Confidentiality – Good for the Reviewers

Let’s say I review a proposal. Let’s say I think it’s absolutely horrible and I detail an enormous numbers of flaws in it and I submit that to the panel. Now let’s say that my review is made public.

Remember: These are my peers. Of the proposals I’ve been asked to review, I know pretty much at least one person on the proposal personally (I’ve met them before or worked with them, or something like that). So from the start, it could obviously hurt any interpersonal relationships.

Remember: These are my peers. In science, we have peer-review before anything gets published. Let’s say that that person whose proposal I panned is a reviewer on a paper I submit a year later, or even an editor at the journal I submitted the paper to. Even though everyone usually tries to be as objective as possible, it’s hard to believe that a public criticism of their proposal by me would not affect their review of my paper. Keep in mind that these proposals for NASA research grants are in the neighborhood of $100,000 requests per year, and the grants usually go for 3 or 4 years (my last proposal was for a 3-year, $448,000 grant).

Or let’s say that in three years, I apply to their school or institution for a job. Even if they are not on the search committee or not directly involved in the hiring, they can still easily affect others’ views of me.

Now let’s say that I give the proposal glowing reviews. I tell the panel that if this proposal is funded, it will change our views of the universe in a fundamental way, this person is the next Einstein, huge paradigm shifts, etc. I argue with everyone and adamantly state that it should be funded. And that review is made public.

The same scenarios happen, only in the opposite way. That person would no longer see me for my research, but that I gave them a good review and helped them (maybe) to get money to fund their research. They would be biased towards me as opposed to against me, and may not be able to judge me fairly against other people who may be more qualified for something. Yes, good for me, but not really fair overall.

What this really boils down to is the same thing with why peer-review of research papers is usually anonymous: Politics and personalities. It can be hard to be honest and objective in a review – especially a negative review – when you know that the person is going to see it.

Confidentiality – Good for the Proposers

At this very early point in my career (I’ve had my Ph.D. for about 18 months now), I have written 3 grant proposals as PI (Principal Investigator, meaning that I’m the one in charge, responsible for everything, and probably doing most of the work). I’ve been on 8 proposals as a Co-I though 3 of them were as a grad student when I would have been a PI if allowed (Co-Investigator, meaning I would get money from the proposal and be in charge of doing some work for it). I’ve been on 4 proposals as a Collaborator (meaning that I’ll provide a supporting role or dataset and have agreed to help the PI and Co-I(s) with a specific task if they are funded, but I won’t get any money from it).

In every single case, I was proposing, we were proposing, or they were proposing new research in an area that had not been explored before. Possibly in an area that had never been thought of before. In one example, I was proposing my own model for something that no one had thought of before, and I was seeking funding to carry out data-gathering to see if my model was supported.

When I submitted those proposals, I expected that no one would see them except for the review panel and program director. I expected that they would not in any way be made public, in any stretch of the imagination. Even by some blogger somewhere who let it slip that he had read a proposal by a redacted person at a redacted institution that was going to look at whether a particular type of crater ejecta could form from atmospheric vortices rather than water.

As soon as my novel approach/ideas are out there, someone else could take them or start working on them and beat me to the punch. Maybe that someone knows of a way to do it faster than I, already has a lab set up, an army of students to throw at the project, and a month later could finish it while I’m still waiting to hear back about funding.

In research, it’s publish or perish, it’s a meritocracy, and if someone publishes first, they usually get the credit. And something like this happened to one of my former officemates (it wasn’t from a NASA review panel, it was that someone overheard her talking in a bar during a conference about work she was going to start).

(Note: The model I mentioned three paragraphs up was proposed in the 1970s, so I’m not giving anything away by putting it in there — I’m giving a “for instance.”)

Final Thoughts

Unfortunately, that’s it for this. I may do a post related to this on the issue of Conflicts of Interest, which are generic to the research process and not just grant proposals. But, I’ll of course honor the program manager’s request and not blog about more details of the process.

When we agree to be a reviewer, we have to sign documents stating that after the panel meets, we will destroy all documents we have that relate to it, including deleting all computer files.

Hopefully you can see from the above discussion* that even though I assured the program manager that I would anonymize everything and not discuss specific proposal ideas, that he or she was still uncomfortable with the idea and requested I not blog about it.

 

*I’m sure there are other reasons, as well, for why confidentiality is necessary in these kinds of things. But, those are the main ones that I can think of.

Advertisement