Exposing PseudoAstronomy

April 3, 2013

Is the Scientific Method a Part of Science?


You probably all remember it, and I can almost guarantee that you were all taught it if you went through any sort of standard American education system (with full recognition for my non-USAian readers). It’s called the Scientific Method.

That thing where you start with a question, form a hypothesis, do an experiment, see if it supports or refutes your hypothesis, iterate, etc. This thing:

Flow Chart showing the Scientific Method

Flow Chart showing the Scientific Method

The question is, does anyone outside of Middle and High School science class actually use it?

A Science Fair Question

I recently judged a middle and high school science fair here in Boulder, CO (USA). The difference in what you see between the two, at least at this science fair, is dramatic: High schoolers are doing undergraduate-level (college) work and often-times novel research while middle schoolers are doing things like, “Does recycled paper hold more weight than non-recycled?” High schoolers are presenting their work on colorful posters with data and graphs and ongoing research questions, while middle schoolers have a board labeled with “Hypothesis,” “Method,” “Data,” and “Conclusions.”

I was asked by a member of the public, after I had finished judging, why that was. He wanted to know why the high school students seemed to have forsaken the entire process and methodology of science, not having those steps clearly laid out.

My answer at the time – very spur-of-the-moment because he was stuttering and I had to catch a bus – was that it IS there in the high school work, but it was more implicit than explicit. That often in research, we have an idea of something and then go about gathering data for it and see what happens: It’s more of an exploration into what the data may show rather than setting out on some narrow path.

That was about a month ago, and I haven’t thought much more about it. But, the Wired article today made me think this would be a good topic for a blog post where I could wax philosophical a bit and see where my own thoughts lay.


A disclaimer up-front (in-middle?) is that I’m an astronomer (planetary geophysicist?). This might be field-specific. The Wired article even mentions astronomy in its list of obvious cases where the Scientific Method is usually not used:

Look at just about any astronomy “experiment”. Most of the cool things in astronomy are also discovered and then a model is created. So, the question comes second. How do you do a traditional experiment on star formation? I guess you could start with some hydrogen and let it go – right? Well, that might take a while.

That said, I’m sure that other fields have the same issues, and it’s really just a big grey area. What I’m going to talk about, that is. Some fields may be more towards one end of the greyscale than the other.

A Recent Paper I Co-Authored

I recently was a co-author on a paper entitled, “ Distribution of Early, Middle, and Late Noachian cratered surfaces in the Martian highlands: Implications for resurfacing events and processes.” The paper was probably the only professional paper I have ever been an author on that explicitly laid out Hypotheses, tests for those hypotheses, what the conclusion would be depending on the results, then the Data, then the Conclusions. And it was a really good way to write THAT paper. But not necessarily other papers.

A Recent Paper I Wrote

I had a paper that was recently accepted (too recently to supply a link). The paper was about estimating and modeling the ages of the largest craters on Mars. There was an Introduction, Methods, Data, and Conclusions. There was no Hypothesis. It was effectively a, “Here is something we can explore with this database, let’s do it and put these numbers out there and then OTHER people may be able to do something with those numbers (or we can) in future work.” There really was no hypothesis to investigate. Trying to make one up to suit the Scientific Method would have been contrived.

This is also something the Wired article mentions:

… often the results of a scientific study are often presented in the format of the scientific method (even though it might not have been carried out in that way). This makes it seem like just about all research in science follows the scientific method.

This is especially the case in medical journals, but not necessarily elsewhere.

Change the “Scientific Method?”

The Wired article offers this as the “new” method:

New Scientific Method (via Wired)

Here’s the accompanying justification:

There are a lot of key elements, but I think I could boil it down to this: make models of stuff. Really, that is what we do in science. We try to make equations or conceptual ideas or computer programs that can agree with real life and predict future events in real life. That is science.

I will preface this next part by saying I am NOT up-to-date on the latest pedagogy of teaching and I am NOT trained in teaching methods (other than 50+ hours of Graduate Teacher Program certification during grad school plus teaching several classes, including two as instructor of record).

That in mind, I think that this is a good idea in later years of grade school education. In the early years, I think that the methodology of the Scientific Method helps get across the basic idea and concepts of how science works, while later on you can get to how it practically works.

Let me explain with an example: In third grade, I was taught about the planets in the solar system plus the sun, plus there are asteroids, plus there are random comets. In eighth grade, I was taught a bit more astronomy and the solar system was a bit messier, but still we had those nine planets (this was pre-2006) and the sun and comets and asteroids plus moons and rings.

Then you get into undergrad and grad school, and you learn about streaming particles coming from the sun, that we can be thought of as being in the sun’s outer-most atmosphere. You get taught about magnetic fields and plasmas. Zodiacal light. The Kuiper Belt, Oort Cloud, asteroid resonances, water is everywhere and not just on Earth, and all sorts of other complications that get into how things really work.

To me, that’s how I think the scientific method should be taught. You start with the rigid formality early on, and I think that’s important because at that level you are really duplicating things that are already well known (e.g. Hypothesis: A ping pong ball will fall at the same rate as a bowling ball) and you can follow that straight-forward methodology of designing an experiment, collecting data, and confirming or rejecting the hypothesis. Let’s put it bluntly: You don’t do cutting-edge science in middle school.

In high school — in a high school with good science education — you actually do start to learn more about the details of different ideas and concepts and solid answers are no longer necessarily known. You want to find out, so you might design an experiment after seeing something weird, and then gather data to try to figure out what’s going on.

That’s how science usually works in the real world, and I think it’s a natural progression from the basic process, and I still think that basic process is implicit, if not explicit, in how science is usually done.

I just got back from a major science conference two weeks ago, and I sat through several dozen talks and viewed several hundred poster presentations. I honestly can’t remember a single one that was designed like a middle school science fair with those key steps from the Scientific Method.

Of course, another aspect is that if we get rid of it, we can’t make comics like this that show how it’s “really” done (sorry, I forget where I found this):

How the Scientific Method Really Works

How the Scientific Method Really Works
(click to embiggen)

Final Thoughts

That said, this has been a ~1400-word essay on what I think about this subject. I don’t expect much to change in the near future, especially since – as the Wired article points out – this is firmly entrenched in the textbooks and in Middle School Science Fair How-To guides.

But, I’m curious as to what you think. Do you think the Scientific Method is useful, useless, or somewhere in-between? Do you think it should be taught and/or used in schools? Do you think it should be used in science fairs? Do you think professional scientists should use it more explicitly more often?



  1. Well, in my field (molecular biology/genetics), we do use it often and students use it to shape their proposals, we use it to shape grant proposals ect. I found it interesting that some fields do not (never really thought of it), but in medical science, we do.

    Comment by Keith Jones — April 3, 2013 @ 5:22 am | Reply

    • I was going to mention – but thought the post was running long – that this *IS* how I framed the last grant I wrote. I even used that language: I have a hypothesis, I’m going to gather these data, and if they show X, it will mean A for my hypothesis, if they show Y, it will mean B for my hypothesis. And, the data I gather will be useful for others.

      I thought the proposal was stronger for that because you really have to show the panel that you know what you’re doing and following that cookie-cutter approach is really useful for it.

      Comment by Stuart Robbins — April 3, 2013 @ 11:06 am | Reply

  2. Useful. Yes, yes, and maybe.

    It’s a bit like cooking, as I see it. Formal recipes exist for good reasons, but expert chefs don’t always follow them exactly. The results are usually good.

    Comment by Expat — April 3, 2013 @ 10:26 am | Reply

  3. Not being a scientist, it was good for me to find out that as scientists become more experienced in their work, the less they need to rely on exactly following the obvious steps of the Scientific Method. I’ve heard of models for quite a while now, especially in the meteorological and astronomical fields, so using those is obviously the way to go, since we now have computers that can do a huge amount of the number crunching for us.

    Probably wouldn’t hurt to write some of the abstracts to read slightly more like the basic method, so lay people like me can follow along a bit better. This would be very helpful with stuff that might turn out controversial, but it’s hard to tell what will set folks off into paroxysms of science denial…

    Anyway, thanks for this post. Very enlightening!

    Comment by Rick K. — April 3, 2013 @ 3:33 pm | Reply

  4. Not so much to disagree with you Stuart, but what about surveys, of which there is an abundance in astronomy (and many other fields)?

    Yes, massive surveys such as SDSS (Sloan Digital Sky Survey) certainly have lots of words about what the results of the survey *might* be used for – hence a back door for hypotheses, etc – but really they’re mostly “I wonder what I’ll find if I take a good hard look at …?”

    Can you spot the ghost of The Scientific Method(TM) in such surveys?

    Comment by Jean Tate — April 5, 2013 @ 6:25 am | Reply

    • I have not read the original proposal for the SDSS, but I would guess that some very strong science cases were made for it to be funded. Same goes for HST — I think there were originally something like 6 fundamental questions that it was designed to answer upon launch. It’s done so much more than that, but it’s those basic Hypothesis (or model or question), Observations to test or explain that, and then Conclusions depending on the result.

      Same goes, really, for my thesis research, creating a global crater database for Mars of roughly 640,000 craters. Yes, a huge survey task, and yes, a HUGE amount of data for OTHER potential studies. But, to get that grant (well, those two grants) funded, I had to have a compelling science case from my end of the handful of questions that I was going to answer/test.

      And that’s why a similar proposal I put in for a comparable lunar crater database last year failed: One of the primary weaknesses identified was that it seemed to be a basic survey project in scope and I needed to make a more compelling science case of what important science questions I was going to answer.

      This isn’t to say I’m trying to pigeon-hole this stuff into the Scientific Method®™. But it gets to my point of that even when it seems as though the SM wasn’t in effect, and though many don’t explicitly use it, the basic process of it is can usually still be found buried within the work.

      Comment by Stuart Robbins — April 5, 2013 @ 11:26 am | Reply

      • Hmm, I think I may not have expressed what I mean clearly enough.

        Yes indeed, proposed astronomical surveys will certainly need a very strong science case before they get funded. And part of those science cases, there will – of course – be all sorts of nice words about how the survey results will/can/might be used to test all sorts of interesting (astrophysical) hypotheses. And certain design aspects of proposed surveys will most definitely be strongly motivated by ease of testing one or other hot topic/hypotheses de jour.

        Whatever. The central motivation, and likely the key reason behind a “GO” decision, will be simple curiosity, a desire to know “what’s there”.

        Look at the history of astronomy in regions of the electromagnetic spectrum inaccessible from the ground, from gammas, to x-rays, to UV; from mid-IR to FIR: why were survey missions funded (ROSAT, IRAS, INTEGRAL, GALEX, even WISE)? I contend the primary reason was that this was, at the time, terra igcognita, an unexplored region. I am quite unfamiliar with similar things in other fields; perhaps exploration of deep sea trenches? or the deeply buried lakes in Antarctica? or partly why the Wollemi Pine (one of very few ‘living fossil’ tree species) was discovered?

        In other words, sure The Scientific Method(TM) may have been invoked, and to an outsider this may seem impressive; however, this was – to turn up the contrast – just window dressing, doing whatever was needed to get the funding/resources (akin to Columbus’ “new trade routes”?).

        Comment by Jean Tate — April 5, 2013 @ 2:31 pm

      • I don’t disagree. It somewhat falls into the, “I see something interesting, let’s learn more about it” category that I attempted to talk about in the post.

        Comment by Stuart Robbins — April 7, 2013 @ 12:39 pm

  5. Excellent lecture, Professor.

    Comment by Jennifer — April 7, 2013 @ 9:04 am | Reply

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: