Why basic research methodology is important to magical knowledge, by Lupa

Lupa in the woods

“If you are going to claim that magic can be proven through experimentation, then your methodology needs to not be half-assed.”

Recently on Livejournal I wrote a response to a post someone else wrote about proposed experiments to try to “prove” the objective existence of Otherkin. These experiments ranged from Kirlian photography to try to get pictures of phantom limbs, to using EEG to measure any neurological abnormalities in Otherkin compared to the general population.

I feel it applies not only to proving Otherkin as something other than collective imagination, but also proving the objective existence of magic.

Research Methodology 101

Here’s what I wrote (with a couple of minor edits and some helpful links added):

With regards to experiments, most of the proposed quantitative experiments over time have been horribly flawed and have not been designed with solid research methodology. Here are a few particular potential flaws:

  • Poor research design: A good piece of research starts with good design. What is the experiment meant to measure? How is it measured? Is it using any existing instruments, or is one created specifically for the purpose of that experiment? Is the instrument you’re using reliable–does it measure consistently? Is it valid – does it measure what you actually are trying to measure? Finally, the simpler, the better, especially in new territory such as this. Keep it to one independent variable and one dependent variable, if possible – and know which is which.
  • Confirmation bias: This is a BIG problem with anecdotal “evidence” of Otherkin, magic, etc. Confimation bias basically means seeing what you want to see, and excluding anything that doesn’t support your desired results. This is often done unconsciously. Example: I keep seeing signs that Tiger is my totem. I want Tiger to be my totem, so I give greater attention and value to things that support Tiger being my totem than not, even though, if the evidence is taken by the numbers, the evidence points toward Tiger not being my totem.
  • Sampling bias: This was a notable reason for why my surveys for the Field Guide were NOT formal research, and a big potential issue with trying to do any experimentation with Otherkin in general. Your sample is most likely going to be biased toward people who A) are willing to be identified in some manner as Otherkin and are not so paranoid as to assume even anonymous research may be used against them personally, and B) more often than not WANT for Otherkin/magic/etc. to be proven. It’s a small population to begin with, too, so you’re most likely going to have a small sample, which can heavily affect whether the research is even solid.
  • Confounds and Correlation vs. Causation: related to some of the earlier things I talked about, confounding variables are variables other than the identified dependent and independent variables that come into play and affect the results. Another, very closely related concept is “correlation does not equal causation”. Just because two variables seem to affect each other in one’s results does not mean that they automatically are causal to each other. There may be a confound or third variable that is the actual vehicle of causation, or the correlation may be coincidence. This is why multiple experiments need to be run, and the results thoroughly analyzed, before making any theoretical conclusions.
  • Applying more significance to results than the statistics show: Statistics are how you analyse your results in various and sundry ways. They allow for a certain level of variation (such as standard deviations from the mean, or identifying outliers) and the statement thereof, and they also help you to rule out whether your results occurred by chance or not (whether your results are statistically significant or not). Through statistics you can use the hard data to determine whether or not you proved your hypothesis (or disproved the null hypothesis).

Because most “evidence” of Otherkin/magic/etc. is anecdotal, and experiments “proving” it often manipulate or inflate the significance of the results, and the best research so far has not supported the objective existence of magic and other spiritual things, any research done to try to “prove” Otherkin/magic/etc. on an objective level needs to be of the highest quality and avoid the above and other pitfalls.

“When I do this, this happens.”

I added one last postscript to my initial response:

(Or, tl;dr – a small handful of people who say “This happens when we do that” does not constitute proper research methodology and does not hold water when trying to prove anything objectively.)

Observing “Well, every time I do this, this happens” is fine if all you want to do is self-confirm a subjective experience. But if you’re trying to prove that magic really works as an independent, objective force (rather than your results being from your own psychological biases, or other external factors that are not “magic”), then you need more rigorous testing then just a handful of people doing the same spell, ritual, or meditation once or twice and comparing their results over coffee.

Just because you claim you can replicate your results doesn’t mean that you can prove that your independent variable and your dependent variable are causative as well as correlated. Are you constructing your experiments with a large enough sample to make a statistical difference? Are you doing your best to rule out confounds and confirmation bias? Would your results hold up to heavy statistical analysis?

The harm of bad research

Every shoddily constructed experiment and instrument, every poorly interpreted or deliberately manipulated set of results, every anecdote held up as firm “evidence” across the board–all these things do absolutely nothing to further your cause, and in fact do much to harm it.

This is one example of what happens when people push bad research into the general consciousness.  And before you say “Well, bad magical research never killed anybody!”, here’s a sizable collection of recorded instances of people being injured or killed by the misapplication of everything from faith healing to dream interpretation (and, apparently, also GPS systems).

Have the correct tools, and be willing to be wrong

And before anyone tries to start a science vs. magic debate, or argue that there’s no such thing as objective reality*, my point that I am making is that if you are going to claim that magic can be proven through experimentation, then your methodology needs to not be half-assed.

If you are going to claim that you have any authority on anything that involves proving something exists objectively, then you need to be literate in the methods used in proving something exists objectively.

Finally, understanding the basics of research methodology is an incredibly valuable part of critical thinking skills, skills that are woefully under-represented in magic and spirituality, and really are a necessary part of being human.**

Those last three paragraphs that I just wrote right up there? THAT’S the intended take-away. You want to prove magic (or any other similar force or concept) exists in an objective, consistently measurable manner? Then have the correct tools, and be willing to be wrong, if that’s where the evidence and statistics end up taking your research.

* I’m not avoiding them because I don’t think they’re good topics of debate, but I want to keep things focused on the actual topic I’m discussing here, rather than getting derailed. Thank you for respecting that.
** Even people who have never, and will never, run a formal experiment still benefit from knowing the basics of research methodology so that they can have a better idea of what the people who do those experiments tell the general public through their published results (and why that’s important to everyday life). Yes, people who are experts in their field and have access to knowledge and training the rest of us don’t do have an advantage and authority. But knowing the basic processes by which they acquire their knowledge, to include research methodology, can help those of us on the general level of “consumer” of information and products to have a better understanding of why, for example, “studies show Brand X is the best!” or parse out whether a news story on “This food/medication/material COULD KILL YOU” is worth paying attention to.

This article was first published at Therioshamanism.com.

The author


Lupa is an author, artist, and neoshaman living in Portland, OR. She earned her Master’s degree in counseling psychology in 2011, with a specific focus on ecopsychology, as a way to integrate the healing and intermediary work of shamanism with a broader cultural and humanistic framework. When she isn’t engaged in creative chaos in her studio, Lupa may be found hiding out in the Columbia River Gorge. Otherwise, she’s online at http://www.thegreenwolf.com and http://therioshamanism.com.

8 Comments on “Why basic research methodology is important to magical knowledge, by Lupa

  1. Pingback: Research Methodology and Paganism? « Confessions of a Hedge Witch

  2. These are important things for people to be aware. Thanks Lupa 🙂

    I would think, in terms of consistent approach, that Humanistic Pagans might consider a re-definition of “magic”. I have redefined “soul”, for example, to mean that complex system of informational memes operating in the brain which forms our individuality and personhood. Meanwhile, some Christian Naturalists define “God” to mean “Reality” and “Revelation” to be what we experience when we learn about reality, most effectively through science.

    If we are to think of these paths as “spiritual languages” then we can rethink what we mean when we use that lexicon. HP’s, for example, do not say “there are no gods”. Instead, they use the word in a manner that is meaningful within their worldview. Likewise, if magic is a critical part of pagan practice, then as modern rational naturalists, I would think HP’s might want to use the word in a manner that is also meaningful and useful.

    Magic has always been about being able to know that which is (cuurently) unknowable and do that which is undoable. We imagine sages spending hours pouring over arcane texts and bubbling cauldrons. We have tried to use it to see remote places, know the future, heal the sick, protect us from harm, manipulate materials, speed our journeys, and more. It seems to me that, to use a Pagan spiritual language, the coming of The Enlightenment (and the scientific method) was not so much a disproving or end to magic. Rather, it was a breakthrough that allowed for the most powerful magic of all to transform the world.

    Today, we *do* have sages studying complex texts and conducting experiments over brews of chemicals. We are able to travel at incredible speeds, heal the sick, make predictions, and so on. This is our magic and it has transformed our world. As for personal magic, I would think that when we use our reason to understand our lives and ourselves better, and use that understanding to achieve happier lives, then we have successfully used our magic. That other stuff could simply be referred to as superstition, false magic, lesser magic, or unproved magic.

    There are many options going forward for Humanistic Pagans, I suppose – I just wanted to share that thought 🙂

  3. Interesting proposal, DT. I fully support finding naturalistic meanings for terms like god, revelation, and magic.

    I do worry, however, that the sort of redefinition mentioned threatens to stretch the integrity of the terms beyond recognition. Science isn’t magic, neither according to scientists nor according to magicians.

    I would like to throw an alternate proposal into the ring. What if, instead of defining science as “magic”, one took current actual magical practices, and simply focused on their naturalistic effects? For example, a love spell might not actually compel another person to love you through some mystical force, but it might bolster your own confidence or at least allay some insecurities. This would seem to maintain more of the integrity of the term “magic.” Same practices, slightly different understanding of them.

    Similarly, what if, instead of defining reality as “god”, one took actual mythic gods and simply looked at what they seem to be from a naturalistic view: cultural entities that play on human nature to provide certain human benefits, like fostering group cohesion and allieving anxiety, in exchange for their replication? What if, instead of defining experience as “revelation”, one simply took actual claimed revelations like scriptures and focused on how these texts inspire and encourage moral behavior?

    To me, these options seem to stretch the meanings less while still delivering fully naturalistic accounts. What’s more, there remains intact a set of concepts and practices that can be followed from a naturalistic path. In my experience, most Pagans who claim literal magical effects with no known physical causal relation, or literal independent deities, also recognize psychological and group benefits as side effects of their practices. A naturalistic path need only focus on the “side effects” as primary effects.

    The kind of redefinition I’m proposing would be analogous to what happened when people realized the sun wasn’t a fiery chariot in the sky, but a star. The term “sun” still referred to the same basic object, though the understanding of the thing had changed.

    In any case, in keeping with the spirit of Lupa’s article, I would also promote ideas about the nature of magic that are *not* naturalistic, provided they are able subjected to the right kind of rigorous research methodology she underlines (and no less). The same goes for my own ideas. Whichever is shown by these methods to be the best explanation, I’d be willing to endorse.

  4. Those are also great points B.T. I like your ideas on this, probably better than my original post. It reminds me of a Shamanistic vision quest I went on once. Although I had a different interpretation of how and why than the supernaturalist folks at the ritual, it had a profound impact on my life and made me realize how useful these kinds of things can be psychologically and in other ways, even as fully natural. Perhaps some new philosophy of naturalistic magic could be developed, which lies between psychology, metaphor, ritual, archetypal issues, etc.

  5. Pingback: Upcoming work « Humanistic Paganism

  6. Playing catch-up now that I’m back in town again; thank you for sharing this here!

    Good commentary. I think as far as redefining magic goes, the problem is that there are several models of magic, some of which conflict. I think we here can agree on the psychological model–for example, a love spell results in the boosting of one’s confidence. But what about the more “supernatural” models–an energy model that relies on an “energy” not (yet) categorized by science, or the spirit model that says that invisible beings make it happen?

    There are existing proponents of the psychological model; chaos magic, for example, is heavily psychological, especially in its earliest forms as per Carroll, Hine, et. al. And we also need to promote the idea that psychology doesn’t have to be dry and bland and rigid; ecopsychology, for example, is a nature-based approach to psychology in theory and practice, and it has a heavily artistic and humanistic-spiritual bent to it.

    So perhaps not so much a full redefinition as an accentuation and building upon what’s already out there?

    • >I think we here can agree on the psychological model–for example, a love spell results in the boosting of one’s confidence. But what about the more “supernatural” models–an energy model that relies on an “energy” not (yet) categorized by science, or the spirit model that says that invisible beings make it happen?

      From an experimental point of view, I think both models are potentially workable, and can be pitted against one another to see which better fits the evidence. First, experiments need to be run to establish that a given magical spell has a statistically significant effect. If that should prove to be the case, then further experiments could be run with relevant modifications. For example, do results persist even when the target has no awareness of the magic (i.e. no possible way of having been psychologically influenced to manifest effects)? If so, do results vary according to the “preferences” of some hypothesized intermediary being (deity or spirit), or are they unaffected? Do results vary according to whether the magician takes a placating or insulting tone to the hypothesized being, indicating consciousness and sociality in the being?

      Such experiments would probably never yield conclusive results, but a lot more could be learned through carefully designed experiments than most people seem to think.

      What Lupa’s article suggests is that any such experiments *cannot be done half-assed.* It would take a rigorous, coordinated effort with close attention paid to rooting out biases and flawed methodology. It may even require formation of a research institution devoted to it. Yet, as intimidating as that may sound, it could vindicate the claims of many magicians. I am a skeptic, obviously, but I would be perfectly willing to reconsider my position in light of such evidence, should it be found. So much stands to be gained, it is a constant marvel to me that it has not been done already.

      Unfortunately, those on the both sides (skeptic and magician) seem all too ready to throw in the towel. Skeptics may point to cases where certain (usually commercial) magicians have been debunked and conclude that all magic is ineffective. Likewise, magicians may wave their hand saying all scientific investigation will inevitably be biased against them, or that magic is inherently inaccessible to science (even though it supposedly has real, measurable effects – a paradox I’ve never understood!). If magic is going to be tested at all, it needs to stop being approached half-assed. That’s what I conclude as the take-away from this article.

  7. Pingback: Where does science fit in? | The Exciting Adventures Beth Barnes: Demonologist Extraordinaire!

%d bloggers like this: