Is the Scientific Method a Part of Science?

Introduction

You probably all remember it, and I can almost guarantee that you were all taught it if you went through any sort of standard American education system (with full recognition for my non-USAian readers). It’s called the Scientific Method.

That thing where you start with a question, form a hypothesis, do an experiment, see if it supports or refutes your hypothesis, iterate, etc. This thing:

Flow Chart showing the Scientific Method

The question is, does anyone outside of Middle and High School science class actually use it?

A Science Fair Question

I recently judged a middle and high school science fair here in Boulder, CO (USA). The difference in what you see between the two, at least at this science fair, is dramatic: High schoolers are doing undergraduate-level (college) work and often-times novel research while middle schoolers are doing things like, “Does recycled paper hold more weight than non-recycled?” High schoolers are presenting their work on colorful posters with data and graphs and ongoing research questions, while middle schoolers have a board labeled with “Hypothesis,” “Method,” “Data,” and “Conclusions.”

I was asked by a member of the public, after I had finished judging, why that was. He wanted to know why the high school students seemed to have forsaken the entire process and methodology of science, not having those steps clearly laid out.

My answer at the time – very spur-of-the-moment because he was stuttering and I had to catch a bus – was that it IS there in the high school work, but it was more implicit than explicit. That often in research, we have an idea of something and then go about gathering data for it and see what happens: It’s more of an exploration into what the data may show rather than setting out on some narrow path.

That was about a month ago, and I haven’t thought much more about it. But, the Wired article today made me think this would be a good topic for a blog post where I could wax philosophical a bit and see where my own thoughts lay.

Field-Specific?

A disclaimer up-front (in-middle?) is that I’m an astronomer (planetary geophysicist?). This might be field-specific. The Wired article even mentions astronomy in its list of obvious cases where the Scientific Method is usually not used:

Look at just about any astronomy “experiment”. Most of the cool things in astronomy are also discovered and then a model is created. So, the question comes second. How do you do a traditional experiment on star formation? I guess you could start with some hydrogen and let it go – right? Well, that might take a while.

That said, I’m sure that other fields have the same issues, and it’s really just a big grey area. What I’m going to talk about, that is. Some fields may be more towards one end of the greyscale than the other.

A Recent Paper I Co-Authored

I recently was a co-author on a paper entitled, “ Distribution of Early, Middle, and Late Noachian cratered surfaces in the Martian highlands: Implications for resurfacing events and processes.” The paper was probably the only professional paper I have ever been an author on that explicitly laid out Hypotheses, tests for those hypotheses, what the conclusion would be depending on the results, then the Data, then the Conclusions. And it was a really good way to write THAT paper. But not necessarily other papers.

A Recent Paper I Wrote

I had a paper that was recently accepted (too recently to supply a link). The paper was about estimating and modeling the ages of the largest craters on Mars. There was an Introduction, Methods, Data, and Conclusions. There was no Hypothesis. It was effectively a, “Here is something we can explore with this database, let’s do it and put these numbers out there and then OTHER people may be able to do something with those numbers (or we can) in future work.” There really was no hypothesis to investigate. Trying to make one up to suit the Scientific Method would have been contrived.

This is also something the Wired article mentions:

… often the results of a scientific study are often presented in the format of the scientific method (even though it might not have been carried out in that way). This makes it seem like just about all research in science follows the scientific method.

This is especially the case in medical journals, but not necessarily elsewhere.

Change the “Scientific Method?”

The Wired article offers this as the “new” method:

New Scientific Method (via Wired)

Here’s the accompanying justification:

There are a lot of key elements, but I think I could boil it down to this: make models of stuff. Really, that is what we do in science. We try to make equations or conceptual ideas or computer programs that can agree with real life and predict future events in real life. That is science.

I will preface this next part by saying I am NOT up-to-date on the latest pedagogy of teaching and I am NOT trained in teaching methods (other than 50+ hours of Graduate Teacher Program certification during grad school plus teaching several classes, including two as instructor of record).

That in mind, I think that this is a good idea in later years of grade school education. In the early years, I think that the methodology of the Scientific Method helps get across the basic idea and concepts of how science works, while later on you can get to how it practically works.

Let me explain with an example: In third grade, I was taught about the planets in the solar system plus the sun, plus there are asteroids, plus there are random comets. In eighth grade, I was taught a bit more astronomy and the solar system was a bit messier, but still we had those nine planets (this was pre-2006) and the sun and comets and asteroids plus moons and rings.

Then you get into undergrad and grad school, and you learn about streaming particles coming from the sun, that we can be thought of as being in the sun’s outer-most atmosphere. You get taught about magnetic fields and plasmas. Zodiacal light. The Kuiper Belt, Oort Cloud, asteroid resonances, water is everywhere and not just on Earth, and all sorts of other complications that get into how things really work.

To me, that’s how I think the scientific method should be taught. You start with the rigid formality early on, and I think that’s important because at that level you are really duplicating things that are already well known (e.g. Hypothesis: A ping pong ball will fall at the same rate as a bowling ball) and you can follow that straight-forward methodology of designing an experiment, collecting data, and confirming or rejecting the hypothesis. Let’s put it bluntly: You don’t do cutting-edge science in middle school.

In high school — in a high school with good science education — you actually do start to learn more about the details of different ideas and concepts and solid answers are no longer necessarily known. You want to find out, so you might design an experiment after seeing something weird, and then gather data to try to figure out what’s going on.

That’s how science usually works in the real world, and I think it’s a natural progression from the basic process, and I still think that basic process is implicit, if not explicit, in how science is usually done.

I just got back from a major science conference two weeks ago, and I sat through several dozen talks and viewed several hundred poster presentations. I honestly can’t remember a single one that was designed like a middle school science fair with those key steps from the Scientific Method.

Of course, another aspect is that if we get rid of it, we can’t make comics like this that show how it’s “really” done (sorry, I forget where I found this):

How the Scientific Method Really Works
(click to embiggen)

Final Thoughts

That said, this has been a ~1400-word essay on what I think about this subject. I don’t expect much to change in the near future, especially since – as the Wired article points out – this is firmly entrenched in the textbooks and in Middle School Science Fair How-To guides.

But, I’m curious as to what you think. Do you think the Scientific Method is useful, useless, or somewhere in-between? Do you think it should be taught and/or used in schools? Do you think it should be used in science fairs? Do you think professional scientists should use it more explicitly more often?

Advertisement

Judging a Middle and High School Regional Science Fair

Introduction

Last week, I judged a regional high school and middle school science fair that was held at my university. I’m trying to be fairly general in my post here and not use specific names, places, etc. in order to keep some semblance of privacy for people involved.

With that in mind, this post contains my experience, thoughts, examples, and general judging criteria, questions I asked, and things that I looked for when judging this fair.

Hopefully it will be (a) an interesting read, and/or (b) useful to future science fair judges, and/or (c) useful for students and parents in preparing for their own science fairs.

Getting Ready

First off, this science fair was for high school (grades 9-12, about ages 13-18) and middle school (grades 6-8, about ages 10-14) students, and it is a “regional” fair, effectively city-wide. It is part of a much larger one where from this, we send people on to state, national, and sometimes just fast-track to international competitions.

There are different categories, in this case the ones I remember were Physics, Earth and Space Science, Environmental, Chemistry, Health and Medicine, I think an Engineering one, and then some others that I don’t remember (’cause they weren’t near my table and I didn’t judge them). When signing up as a judge, we could request the main topic we wanted to judge and then were asked to fill in 1-3 others. I requested Earth and Space Science primarily, and Physics as my second. I also checked the box indicating I wanted to be a “head judge,” meaning that within a particular subject, I would be responsible for determining 1st, 2nd, and 3rd places, and I would also confer with the Roaming Judges about best in show and who to send on to state/national/international. Based on who the head judges were last year and that I had judged this fair before, I thought I had a decent shot.

The week before, I got an e-mail saying I’d been placed in Physics and Environmental. I was not a head judge. When I showed up the morning of, I got my packet and found I was in Earth and Space Science (ESS) in the morning along with Environmental, and ESS and Physics in the afternoon. Still not a head judge. It had been changed because they had last-minute judge cancellations.

The Day Of

I went to sit down before we started at the ESS table. I got to talking with a few people there – only one whom I’d met before – and then the head judge for our table – who studies space weather forecasting – introduced herself and told us what her bias was in judging. She liked the projects where the students really seemed to have a passion for it and followed the scientific method, as evidenced by them coming up with the project themselves, and then forming a hypothesis and working towards answering it.

Fair ‘nough. Since she had told us that, I decided I would say what mine were. I said that I really liked it when the students would form a hypothesis and then their experiment showed that they were actually wrong. And then that they decided that, on the basis of their data, their initial hypothesis was incorrect.

(I said that they get brownie points if it’s a common pseudoscience. This isn’t to say that I would judge students more harshly if they started off with a “correct” hypothesis, just that I think it is admirable when they are willing to change their beliefs on the basis of observable data – since on this blog I have illustrated many cases where people will not.)

Then I gave an example from the previous year of a middle schooler who thought that granite was radioactive and emitted poison radon gas. He found out by testing various common building rocks that he was wrong and he changed his mind about the initial hypothesis as a result. I then gave a second example of a high school student last year who thought that magnetic healing was real and that she …

That was where I was interrupted by the head judge. She said, “But magnetic healing does work.”

I looked at her and replied, “I mean stuff like the bracelets …” and that was when she held up her wrist and I saw one on her “… because blood is made with non-ferromagnetic iron.”

While she admitted that it may be a placebo effect, she claimed that she had debilitating arthritis in her wrist and that the magnets in the bracelet really helped her to function. I dropped it and moved on. But that was how my morning started. I figure if I had been in the Health and Medicine category, I would’ve spoken with the organizer at that time.

Brief Overview of the Day

In the morning session, the high schoolers are judged by up to 6 or 7 judges to get them more exposure. In the afternoon, each middle schooler is judged by up to 4 or 5.

Anyway, the next 8 hours of judging were fairly uneventful. There were some pretty good projects, and there were some really bad ones. Since I’ve seen people ask what judges actually look for and not just what we’re told to look for, I thought I’d post some specific examples of good and bad:

Example of a BAD Middle School Project

There was a middle school project where two guys wanted to see how quickly they could turn a water wheel with a hose based upon how far above the wheel the hose was as some sort of analogy for a hydroelectric dam. One of the guys was out sick, so he may have been the smrt one in the group (see what I did there?).

Besides not really being comparable to most of the projects there in terms of science nor skill, they got the “wrong” answer despite having the “right” physics on the poster and not realizing it. What I mean is that they found their wheel spun faster when the hose was closer to it. He said they had issues with the water blowing in the breeze when they held it high up which may have affected it. So I asked how it did not mimic an actual dam, and the guy thought, and then said the height of the water. I asked, “What else?” He didn’t know … I was looking for the idea that real dams aren’t bothered by the wind.

In his talk to me, he had mentioned potential energy, so I asked him what it was, trying to get him to realize the “right” answer for the overall project. He said he didn’t know, he hadn’t been paying attention in class that day. I looked pointedly at a location on his poster, then he looked, and said, “Oh, it’s right there …” he proceeded to read what they had written as to the definition of potential energy, and then just looked at me and said, “Yeah.”

(For those wondering, the higher the water starts, the more potential energy it has to convert to kinetic energy to spin the wheel.)

Example of a BAD High School Project

A bad one I judged at the high school level was a girl who was looking at minor gas components to Earth’s atmosphere and their affects on infrared (IR) absorption -> greenhouse effects. She used basically a blackbox model (she had no idea what went into it, it was a computer program handed to her) and used Earth’s standard atmosphere.

She modeled Earth and the sun as two black bodies. When I asked her to explain black bodies to me, she pointed to a part of her poster that had the equation and basically read the equation to me. I asked her what “I” was (as in the equation is “I = …”). She rambled off the standard definition of radiative energy per square unit per solid angle per blah blah blah. I asked her to tell me what it was in her own words. She didn’t know. I asked if, “Intensity” could be a synonym, and she said, “I guess.”

One of the results she found was that water vapor accounts for around 60% of IR absorption in the atmosphere. I asked her how much water vapor was in the atmosphere in her model. Since a second step was that she varied the amount from the standard atmospheric composition (making, for example, methane 500% of what it really is), she said, “100%.” I clarified: “No, I mean how much of the atmosphere in the standard model is water vapor?” She didn’t know.

I dinged her quite a bit for (a) just spouting off terms and not knowing what they were, and (b) not knowing what went into her model at all, especially in a field (climate modeling / global warming) where some of the major criticisms are what assumptions and parameters go into the models.

Example of a MEDIOCRE Middle School Project

There was a girl who looked at different soil samples around the county in a North-South then East-West line. She did a decent job, found a pattern or two, and was able to apply it to other things like saying that based on her results, if someone wanted to grow a garden she would recommend certain places and not others. Her stuff was good, but just within the scope of the project. There was nothing really brought in from outside, no bigger picture, and she got some terminology wrong, like what “topography” was. She also had told the judge just before me that she did the project in the space of a week before the Fair because her teacher suggested it to her since she had done well in their soils unit of science class.

Example of a Good Middle School Project

A good middle school project I saw was where a guy had built (supposedly himself) a water tank and made wheels out of styrofoam. The wheels all had the same outside diameter, different inside diameters, and then spokes. It was a model for turbulence for bike wheels.

He threw out terms like “Reynolds Number” and when I asked what it meant, he was able to answer it. Then I asked him if the Reynolds Number of water was larger or smaller than molasses. He got it wrong. Then I asked which had a higher viscosity. He got it right.

Moving on, after he had done his models in water, he tried out different tires with different spoke lengths on a bike — apparently he’s a “pro” biker (remember – a middle schooler) and has won some significant races. He found that the benefits from the water model were much more muted, by say 5% increases in speed due to reduced turbulence instead of the 100% he was seeing in the tank. He was able to answer why that may be. And he was able to say that even at 1-2% when you’re talking about a race where he won by 0.04 seconds over 17 minutes, then that really counts.

Things I Asked and Looked For

1. I started out after introducing myself and telling the student to assume I knew nothing about their project and subject and to start from there. In other words, I really wanted to see if they could explain it to a lay person without using the big words, and in their own words. I also ran into a problem last year where some of the high school projects were over my head (that wasn’t an issue this year); this had been my fault, really, since I should have made them explain it to me until I understood it and could tell if they understood it.

2. Whenever they used a term or concept that I thought was above grade-level, I would stop them and ask them to explain it. Sometimes they could, sometimes they couldn’t. I did count it against them if they couldn’t because it meant they were just using buzz words and didn’t know what they actually meant.

3. About half-way through each project, I would pause and repeat back to them a general summary of what they were doing to make sure that I understood it. After all, it wasn’t fair to the student if I thought they were doing something they weren’t, and I thought it helped the student because it showed I was actually paying attention.

4. I asked them what the bigger picture was at the end. How they could apply it to something else, or what it could do for people? Most of them were able to answer that, and I would hope it’s a fairly standard question. There were admittedly some cases where there was no real practical application, such as determining rotation rates for stars, but in those cases I would ask them what the application to the field would be, instead of every-day life.

5. I would always ask if there was anything else they wanted to tell me or thought I should know before I left and said, “Thank you, and good luck.” I thought that was important in case they realized at that time they had left something important out.

6. If I remembered, I asked them how they got the idea for the project and how much help they had. Unfortunately, I didn’t remember to do that every time. But in general, you can tell as a judge how much help they had had.

7. One aspect that I in particular looked for was confidence in the results – as in error bars. Almost no one had error bars, and the one who did I asked how he got them and whether they were believable.

The error bar obsession probably comes from my own research and my professors from undergrad. In fact, I’m presenting some research on age-dating martian volcanos this week at a conference (LPSC XLI) that’s basically ALL about error bars, and it’s what I expect pretty much everyone to ask me about at my poster on Thursday night (if you’re actually interested in this, ask below in the comments).

An example of the importance of this was a high school student who was doing an experiment growing algae, and he found a negative mass on the 3rd day. I asked him why. He said he didn’t know, maybe something about the atmosphere changing. I asked him what his uncertainty was in his measurements, and he said that when all the analysis was done, he would do the detailed fitting analysis and the uncertainty in his fits. I replied, “No, each measurement you made has an inherent uncertainty in it, and you should be displaying that on your graph. For example, you found a mass of negative 0.01 gms. If your uncertainty in that measurement is ±0.1, then you’re fine. If it’s ±0.00001, then you have a problem.”

In another example, there was a student who I thought was the best in his category that I saw in the middle school part. He had looked at earthquake data in our state in order to test an empirical law, and he showed that it was wrong for small earthquakes. He was graphing magnitude versus number of quakes, and was looking at mag. 3-4 and found that it fit the law with a value of 13, but then 1-2 didn’t fit where he found 9 but expected 60. I asked him what the uncertainty was in each point. He didn’t know, but he said that he found only about 10% of what he expected. I told him I realized that, but in his magnitude 3-4 bin, he found it kinda fit the law, but the question was how well it did. In other words, at what magnitude does the law break-down with statistical “certainty.” He didn’t know. Since I was the last judge for him for the day, I told him he needed to look into Poisson statistics (sometimes referred to as “counting” statistics).

Some of the Formal, Official Stuff, and Judging Paperwork

The way I went about the actual judging part was that I left the score sheets alone. I went to each person with a blank sheet of paper and just wrote down notes. I spent about 10-17 minutes with each person (we were told to spend 12 on middle school, 15 on high school), and I did time it. I felt this was important (a) so the students had an equal amount of time with me, and (b) so that there wasn’t a judge waiting because I was taking up all the student’s time (speaking from experience of being that judge waiting …).

When I was done with each student, I went to a table and wrote down comments on the judging form that the students would actually see. I then went onto the next student I was judging.

I waited until I had talked with ALL students in the section before I sat down to actually assign a numerical score (morning was high school, afternoon was middle school). I figured that was the fairest way to do it so that I wasn’t too easy nor too hard at the beginning of the judging period and so that I could get a feel for the general level of the projects and rate them accordingly.

We were told to judge on a 100-pt scale, and judge independent of whether it was individual or team, and whether they worked on their own or in a lab. 30% went to creative ability, 30% to the scientific thought, 15% to thoroughness, 15% to skill, 10% clarity, and then an additional 16 pts to teamwork where the score was then normalized to 100 for teams. If people are interested in more details on what each of these were defined as, ask me in the Comments section.

Final Thoughts

I find judging – the two times I’ve done it – fairly enjoyable. On a jealous note, it’s really amazing what these young adults can do. When I look back at my middle school and high school science projects, there simply was nothing comparable until I specifically chose my “Senior Project” at the end of high school and built a model roller coaster.

Most of the projects at least show some interest in their chosen topic, especially at the high school level. There are always those few where you can tell that the student felt obligated to do something they really don’t care about, but luckily those can be offset by the ones that show thought.

Determining how much help students had is sometimes hard, but it’s important because it can mean the difference between someone winning first place or not even being in the running (something we did last year on a particular tennis racket project). It can be a hard judgement call, but it’s one that I have given my input on and then left it up to the head judge in the topic to decide.

Anyway, to sum up, I hope that this has been interesting and/or informative. If you’re a scientist, I highly recommend judging a science fair, at least once, to get a feel for it and to try to encourage young, potentially future scientists. Those that are actually interested in what they’re doing do enjoy speaking with people who are not their parents/advisor/mentor that understand what they’re doing and its importance.