Is the Scientific Method a Part of Science?

Introduction

You probably all remember it, and I can almost guarantee that you were all taught it if you went through any sort of standard American education system (with full recognition for my non-USAian readers). It’s called the Scientific Method.

That thing where you start with a question, form a hypothesis, do an experiment, see if it supports or refutes your hypothesis, iterate, etc. This thing:

Flow Chart showing the Scientific Method

The question is, does anyone outside of Middle and High School science class actually use it?

A Science Fair Question

I recently judged a middle and high school science fair here in Boulder, CO (USA). The difference in what you see between the two, at least at this science fair, is dramatic: High schoolers are doing undergraduate-level (college) work and often-times novel research while middle schoolers are doing things like, “Does recycled paper hold more weight than non-recycled?” High schoolers are presenting their work on colorful posters with data and graphs and ongoing research questions, while middle schoolers have a board labeled with “Hypothesis,” “Method,” “Data,” and “Conclusions.”

I was asked by a member of the public, after I had finished judging, why that was. He wanted to know why the high school students seemed to have forsaken the entire process and methodology of science, not having those steps clearly laid out.

My answer at the time – very spur-of-the-moment because he was stuttering and I had to catch a bus – was that it IS there in the high school work, but it was more implicit than explicit. That often in research, we have an idea of something and then go about gathering data for it and see what happens: It’s more of an exploration into what the data may show rather than setting out on some narrow path.

That was about a month ago, and I haven’t thought much more about it. But, the Wired article today made me think this would be a good topic for a blog post where I could wax philosophical a bit and see where my own thoughts lay.

Field-Specific?

A disclaimer up-front (in-middle?) is that I’m an astronomer (planetary geophysicist?). This might be field-specific. The Wired article even mentions astronomy in its list of obvious cases where the Scientific Method is usually not used:

Look at just about any astronomy “experiment”. Most of the cool things in astronomy are also discovered and then a model is created. So, the question comes second. How do you do a traditional experiment on star formation? I guess you could start with some hydrogen and let it go – right? Well, that might take a while.

That said, I’m sure that other fields have the same issues, and it’s really just a big grey area. What I’m going to talk about, that is. Some fields may be more towards one end of the greyscale than the other.

A Recent Paper I Co-Authored

I recently was a co-author on a paper entitled, “ Distribution of Early, Middle, and Late Noachian cratered surfaces in the Martian highlands: Implications for resurfacing events and processes.” The paper was probably the only professional paper I have ever been an author on that explicitly laid out Hypotheses, tests for those hypotheses, what the conclusion would be depending on the results, then the Data, then the Conclusions. And it was a really good way to write THAT paper. But not necessarily other papers.

A Recent Paper I Wrote

I had a paper that was recently accepted (too recently to supply a link). The paper was about estimating and modeling the ages of the largest craters on Mars. There was an Introduction, Methods, Data, and Conclusions. There was no Hypothesis. It was effectively a, “Here is something we can explore with this database, let’s do it and put these numbers out there and then OTHER people may be able to do something with those numbers (or we can) in future work.” There really was no hypothesis to investigate. Trying to make one up to suit the Scientific Method would have been contrived.

This is also something the Wired article mentions:

… often the results of a scientific study are often presented in the format of the scientific method (even though it might not have been carried out in that way). This makes it seem like just about all research in science follows the scientific method.

This is especially the case in medical journals, but not necessarily elsewhere.

Change the “Scientific Method?”

The Wired article offers this as the “new” method:

New Scientific Method (via Wired)

Here’s the accompanying justification:

There are a lot of key elements, but I think I could boil it down to this: make models of stuff. Really, that is what we do in science. We try to make equations or conceptual ideas or computer programs that can agree with real life and predict future events in real life. That is science.

I will preface this next part by saying I am NOT up-to-date on the latest pedagogy of teaching and I am NOT trained in teaching methods (other than 50+ hours of Graduate Teacher Program certification during grad school plus teaching several classes, including two as instructor of record).

That in mind, I think that this is a good idea in later years of grade school education. In the early years, I think that the methodology of the Scientific Method helps get across the basic idea and concepts of how science works, while later on you can get to how it practically works.

Let me explain with an example: In third grade, I was taught about the planets in the solar system plus the sun, plus there are asteroids, plus there are random comets. In eighth grade, I was taught a bit more astronomy and the solar system was a bit messier, but still we had those nine planets (this was pre-2006) and the sun and comets and asteroids plus moons and rings.

Then you get into undergrad and grad school, and you learn about streaming particles coming from the sun, that we can be thought of as being in the sun’s outer-most atmosphere. You get taught about magnetic fields and plasmas. Zodiacal light. The Kuiper Belt, Oort Cloud, asteroid resonances, water is everywhere and not just on Earth, and all sorts of other complications that get into how things really work.

To me, that’s how I think the scientific method should be taught. You start with the rigid formality early on, and I think that’s important because at that level you are really duplicating things that are already well known (e.g. Hypothesis: A ping pong ball will fall at the same rate as a bowling ball) and you can follow that straight-forward methodology of designing an experiment, collecting data, and confirming or rejecting the hypothesis. Let’s put it bluntly: You don’t do cutting-edge science in middle school.

In high school — in a high school with good science education — you actually do start to learn more about the details of different ideas and concepts and solid answers are no longer necessarily known. You want to find out, so you might design an experiment after seeing something weird, and then gather data to try to figure out what’s going on.

That’s how science usually works in the real world, and I think it’s a natural progression from the basic process, and I still think that basic process is implicit, if not explicit, in how science is usually done.

I just got back from a major science conference two weeks ago, and I sat through several dozen talks and viewed several hundred poster presentations. I honestly can’t remember a single one that was designed like a middle school science fair with those key steps from the Scientific Method.

Of course, another aspect is that if we get rid of it, we can’t make comics like this that show how it’s “really” done (sorry, I forget where I found this):

How the Scientific Method Really Works
(click to embiggen)

Final Thoughts

That said, this has been a ~1400-word essay on what I think about this subject. I don’t expect much to change in the near future, especially since – as the Wired article points out – this is firmly entrenched in the textbooks and in Middle School Science Fair How-To guides.

But, I’m curious as to what you think. Do you think the Scientific Method is useful, useless, or somewhere in-between? Do you think it should be taught and/or used in schools? Do you think it should be used in science fairs? Do you think professional scientists should use it more explicitly more often?

Advertisement

Skeptiko Host Alex Tsakiris: On the Non-Scientifically Trained Trying to Do/Understand Science

Preamble

First, let me give one announcement for folks who may read this blog regularly (hi Karl!). This may be my last post for about a month or so. As you may remember from my last post, I will be teaching all next month, June 1 through July 2, and the class is every day for 95 minutes. I have no idea how much free time I may have to do a blog post, and I have some other projects I need to finish up before the end of the month (I’m also a photographer and I had a bride finally get back to me about photos she wants finished).

Introduction

I have posted once before about Skeptiko podcast host Alex Tsakiris in my post about The Importance of Peer-Review in Science. The purpose of that post was to primarily show that peer review is an important part of the scientific process, a claim contrary to what the host of said podcast had claimed.

Now for the official disclaimer on this post: I do not know if Alex is a trained scientist. Based on what he has stated on his podcast, my conclusion is that he is not. What I have read of his background (something like “successful software entrepreneur” or around those lines) supports that conclusion. However, I don’t want to be called out for libel just in case and so that is my disclaimer.

Also, I am not using this post to say whether I think near-death experiences are a materialistic phenomenon or point to a mind-brain duality (mind/consciousness can exist separately from brain). That is NOT the point of this post and I am unqualified to speak with any authority on the subject (something I think Alex needs to admit more often).

Anyway, I just completed listening to the rather long Skeptiko episode #105 on near-death experiences with Skeptics’ Guide to the Universe host Steven Novella Dr. Steven Novella (see Points 2 and 3 below for that “Dr.” point). I want to use that episode to make a few points about how science is done that an (apparently) non-scientifically-trained person will miss. This post is not meant to be a dig/diss against so-called “citizen science,” rather the pitfalls of which non-scientists should be aware when trying to investigate pretty much ANY kind of science.

Point 1: Conclusions Are Not Data

Many times during the episode’s main interview and after the interview in the “follow-up,” Alex would talk about a paper’s conclusions. “The researchers said …” was a frequent refrain, or “In the paper’s conclusions …” or even “The conclusions in the Abstract …” I may be remembering incorrectly, perhaps someone may point that out, but I do not recall any case where Alex instead stated, “The data in this paper objectively show [this], therefore we can conclude [that].”

This is a subtle difference. Those of you who may not be scientifically trained (or listened to Steve’s interview on the episode) may not notice that there is an important (though subtle) difference there. The difference is that the data are what scientists use to make their conclusion. A conclusion may be wrong. It may be right. It may be partially wrong and partially right (as shown later on with more studies … more data). Hopefully, if there was not academic fraud, intellectual dishonesty, nor faulty workmanship (data gathering methods), the actual data itself will NEVER be wrong, just the conclusions from it. In almost any paper — at least in the fields with which I am familiar — the quick one-line conclusions may be what people take away and remember, but it’s the actual data that will outlive that paper and that other researchers will look at when trying to replicate, use in a graduate classroom, or argue against.

I will provide two examples here, both from my own research. The first is from a paper that I just submitted on using small, 10s to 100s meter-sized craters on Mars to determine the chronology of the last episodes of volcanism on the planet. In doing the work, there were only one or two people who had studied it previously, and so they were obviously talked about in my own paper. Many times I reached the same conclusion as they in terms of ages of some of the volcanos, but several times I did not. In those cases, I went back to their data to try to figure out where/why we disagreed. It wasn’t enough just to say, “I got an age of x, she got an age of y, we disagree.” I had to look through and figure out why, and whether we had the same data results and if so why our interpretations differed, or if our actual data differed.

The second example that’s a little better than the first is with a paper I wrote back in 2008 and was finally published in a special edition of the journal Icarus in April 2010 (one of the two main planetary science journals). The paper was on simulations I did of Saturn’s rings in an attempt to determine the minimum mass of the rings (which is not known). My conclusion is that the minimum mass is about 2x the mass inferred from the old Voyager data. That conclusion is what will be used in classrooms, I have already seen used in other peoples’ presentations, and what I say at conferences. However, people who do research on the rings have my paper open to the data sections, and I emphasize the “s” because in the paper, the data sections (plural) span about 1/2 the paper, the methods section spans about 1/3, and the conclusions are closer to 1/6. When I was doing the simulations, I worked from the data sections of previous papers. It’s the data that matters when looking at these things, NOT an individual (set of) author(s).

Finally for this point, I will acknowledge that Alex often repeats something along the lines of, “I just want to go where the data takes us.” However, saying that and then reading a paper’s conclusions are not mutually compatible. Steve pointed that out at least twice during the interview. At one point in the middle, he exclaimed (paraphrasing), “Alex, I don’t care what the authors conclude in that study! I’m looking at their data and I don’t think the data supports their conclusions.”

Point 2: Argument from Authority Is Not Scientific Consensus

In my series that I got about half-way through at the end of last year on logical fallacies, I specifically avoided doing Argument from Authority because I needed to spend more time on it versus the Scientific Consensus. I still intend to do a post on that, but until then, this is the basic run-down: Argument from Authority is the logical fallacy whereby someone effectively states, “Dr. [so-and-so], who has a Ph.D. in this and is well-credentialed and knows what they’re doing, says [this], therefore it’s true/real.”

If any of my readers have listened to Skeptiko, you are very likely familiar with this argument … Alex uses it in practically EVERY episode MULTIPLE times. He will often present someone’s argument as being from a “well-credentialed scientist” or from someone who “knows what they’re doing.” This bugs the — well, this is a PG blog so I’ll just say it bugs me to no end. ALL BECAUSE SOMEONE HAS A PH.D. DOES NOT MEAN THEY KNOW WHAT THEY’RE DOING. ALL BECAUSE SOMEONE HAS DONE RESEARCH AND/OR PUBLISHED DATA DOES NOT MEAN THEIR CONCLUSIONS ARE CORRECT OR THAT THEY GATHERED THEIR DATA CORRECTLY.

Okay, sorry for going all CAPS on you, but that really cannot be said enough. And Alex seems to simply, plainly, and obviously not understand that. It is clear if you listen to practically any episode of his podcast, especially during any of the “psychic dogs” episodes or “global consciousness” ones. It was also used several times in #105, including one where he explicitly stated that a person was well-credentialed and therefore knows what they’re doing.

Now, very briefly, a single argument from someone does not a scientific consensus make. I think that’s an obvious point, and Steve made it several times during the interview that there is no consensus on the issue and individual arguments from authority are just that — arguments from authority and you need to look at their data and methods before deciding for yourself whether you objectively agree with their conclusions.

Edited to Add: I have since written a lengthy post on the argument from authority versus scientific consensus that I highly recommend people read.

Point 3: Going to Amazon, Searching for Books, to Find Interview Guests

Okay, I’ll admit this has little to do with the scientific process on its face, but it illustrates two points. First, that Alex doesn’t seem to understand the purpose/point of scientific literature, and second that fast-tracking the literature and doing science by popular press is one of the worst ways and a way that strikes many “real” scientists as very disingenuous. I’ll explain …

First, I will again reference my post, “The Importance of Peer-Review in Science.” Fairly self-explanatory on the title, and I will now assume that you’re familiar with its arguments. In fact, I just re-read it (and I have since had my own issues fighting with a reviewer on a paper before the journal editor finally just said “enough” and took my side).

To set the stage, Alex claims in the episode:

“Again, my methodology, just so you don’t think I’m stacking the deck, is really simple. I just go to Amazon and I search for anesthesia books and I just start emailing folks until one of them responds.”

As I explained, peer-reviewed papers are picked apart by people who study the same thing as you do and are familiar with other work in the area. A book is not. A book is read by the publishing company’s editor(s) – unless it’s self-published in which case it’s not even read by someone else – and then it’s printed. There is generally absolutely zero peer-review for books, and so Alex going to Amazon.com to find someone who’s “written” on the subject of near-death experiences will not get an accurate sampling. It will get a sampling of people who believe that near-death experiences show mind-brain duality because …

Published books on a fringe “science” topic are done by the people who generally have been wholeheartedly rejected by the scientific community for their methods, their data-gathering techniques, and/or their conclusions not being supported by the data. But they continue to believe (yes, I use the word “believe” here for a reason) that their interpretations/methods/etc. are correct and hence instead of learning from the peer-review process and tightening their methods, trying to bring in other results, and looking at their data in light of everything else that’s been done, they publish a book that simply bypasses the last few steps of the scientific process.

Not to bring in politics, but from a strictly objective point, this is what George W. Bush did with the US’s “missile defense” system. Test after test failed and showed it didn’t work. Rather than going back and trying to fix things and test again, he just decided to build the thing and stop testing.

Point 4: Confusing a Class of Outcomes with a Single Cause

This was more my interpretation of what Alex did in the interview and what Steve pointed out at many times, and it is less generalizable to the scientific process, but it does apply nonetheless.

Say, in cooking, you serve up a pizza. The pizza is the “class of experiences” here that is the same as a class of things that make up the near-death experience (NDE). The toppings of your pizza are the individual experiences of the NDE. Pizzas will usually have cheese, NDEs will usually have a sense of well-being. Pizzas may more rarely have onions, NDEs may more rarely have a white light tunnel associated with them. You get the idea.

Now, from the impression I got, Alex seemed to claim throughout the episode that there was only one way to make a pizza — have an NDE. Steve argued that there were many different ways to make a pizza, and that all those different techniques will in general lead to something that looks like a pizza.

Point 5: Steve’s a Neurologist, Alex Is Not

I need to say before I explain this point that I am NOT trying to say that you need a Ph.D. in the topic to do real science. I do not in ANY WAY mean to imply that science is an elitist thing where only people “in the club” can participate.

That said, I really am amazed by Alex arguing against people who actually have studied the subject for decades. If you are a non-scientist, or even if you are a scientist but have not studied the topic at-hand (like, gee, me talking about near-death experiences while I’m an astrophysicist/geophysicist), then you need to make darn sure that you know what the heck you’re talking about. And you need to be humble enough to, when the actual person who’s studied this says you’ve made a mistake, take that very seriously and look again at what you thought was going on. The probability that you have made a mistake or misunderstood something as opposed to the expert in the field is fairly high.

Again, this is not my attempt to backtrack and myself commit an argument from authority fallacy. However, there is a difference from making an argument from authority fallaciously versus listening to what an authority on the subject says and taking it into account and re-examining your conclusions. It seriously amazes me how much Alex argued against Steve as if Alex were an expert in neurology. It caused him to simply miss many of the points and arguments Steve was making, as evidenced by Steve saying something and then needing to repeat his argument 20 minutes later because Alex had ignored it because Alex has been buoyed by his interviews with previous pro-duality guests.

Final Thoughts

As I’ve stated, the purpose of this post is not to discuss whether NDEs show a mind-brain duality or if it has a purely materialistic explanation. The purpose is to point out that the methods Alex uses are fallacious, and while I know that people have pointed it out to him before, it seems that it has made very little impact upon the way he argues. I believe this is in part due to his need for confirmation bias – he definitely has made his mind up on whether or not psi-type phenomena exists. But I also am fairly sure that it’s because Alex lacks any kind of formal training in science. Because of that, he makes these kinds of mistakes – at least originally – without knowing any better. Now, since it’s been pointed out to him, I think it’s intellectually dishonest to keep making them, but again that’s beyond the purpose of this post.

So, to wrap this all up, non-scientists take heed! Avoid making these kinds of mistakes when you try to do or to understand science yourself. Make sure that you look at the data, not just the conclusions from a paper. Don’t make arguments from authority. Remember that popular books are not the same as peer-reviewed literature. And keep in mind there can be (a) multiple explanations and (b) multiple ways to reach an end point.