Naturalistic Paganism

“If you are going to claim that magic can be proven through experimentation, then your methodology needs to not be half-assed.”

Recently on Livejournal I wrote a response to a post someone else wrote about proposed experiments to try to “prove” the objective existence of Otherkin. These experiments ranged from Kirlian photography to try to get pictures of phantom limbs, to using EEG to measure any neurological abnormalities in Otherkin compared to the general population.

I feel it applies not only to proving Otherkin as something other than collective imagination, but also proving the objective existence of magic.

Research Methodology 101

Here’s what I wrote (with a couple of minor edits and some helpful links added):

With regards to experiments, most of the proposed quantitative experiments over time have been horribly flawed and have not been designed with solid research methodology. Here are a few particular potential flaws:

• Poor research design: A good piece of research starts with good design. What is the experiment meant to measure? How is it measured? Is it using any existing instruments, or is one created specifically for the purpose of that experiment? Is the instrument you’re using reliable–does it measure consistently? Is it valid – does it measure what you actually are trying to measure? Finally, the simpler, the better, especially in new territory such as this. Keep it to one independent variable and one dependent variable, if possible – and know which is which.
• Confirmation bias: This is a BIG problem with anecdotal “evidence” of Otherkin, magic, etc. Confimation bias basically means seeing what you want to see, and excluding anything that doesn’t support your desired results. This is often done unconsciously. Example: I keep seeing signs that Tiger is my totem. I want Tiger to be my totem, so I give greater attention and value to things that support Tiger being my totem than not, even though, if the evidence is taken by the numbers, the evidence points toward Tiger not being my totem.
• Sampling bias: This was a notable reason for why my surveys for the Field Guide were NOT formal research, and a big potential issue with trying to do any experimentation with Otherkin in general. Your sample is most likely going to be biased toward people who A) are willing to be identified in some manner as Otherkin and are not so paranoid as to assume even anonymous research may be used against them personally, and B) more often than not WANT for Otherkin/magic/etc. to be proven. It’s a small population to begin with, too, so you’re most likely going to have a small sample, which can heavily affect whether the research is even solid.
• Confounds and Correlation vs. Causation: related to some of the earlier things I talked about, confounding variables are variables other than the identified dependent and independent variables that come into play and affect the results. Another, very closely related concept is “correlation does not equal causation”. Just because two variables seem to affect each other in one’s results does not mean that they automatically are causal to each other. There may be a confound or third variable that is the actual vehicle of causation, or the correlation may be coincidence. This is why multiple experiments need to be run, and the results thoroughly analyzed, before making any theoretical conclusions.
• Applying more significance to results than the statistics show: Statistics are how you analyse your results in various and sundry ways. They allow for a certain level of variation (such as standard deviations from the mean, or identifying outliers) and the statement thereof, and they also help you to rule out whether your results occurred by chance or not (whether your results are statistically significant or not). Through statistics you can use the hard data to determine whether or not you proved your hypothesis (or disproved the null hypothesis).

Because most “evidence” of Otherkin/magic/etc. is anecdotal, and experiments “proving” it often manipulate or inflate the significance of the results, and the best research so far has not supported the objective existence of magic and other spiritual things, any research done to try to “prove” Otherkin/magic/etc. on an objective level needs to be of the highest quality and avoid the above and other pitfalls.

“When I do this, this happens.”

I added one last postscript to my initial response:

(Or, tl;dr – a small handful of people who say “This happens when we do that” does not constitute proper research methodology and does not hold water when trying to prove anything objectively.)

Observing “Well, every time I do this, this happens” is fine if all you want to do is self-confirm a subjective experience. But if you’re trying to prove that magic really works as an independent, objective force (rather than your results being from your own psychological biases, or other external factors that are not “magic”), then you need more rigorous testing then just a handful of people doing the same spell, ritual, or meditation once or twice and comparing their results over coffee.

Just because you claim you can replicate your results doesn’t mean that you can prove that your independent variable and your dependent variable are causative as well as correlated. Are you constructing your experiments with a large enough sample to make a statistical difference? Are you doing your best to rule out confounds and confirmation bias? Would your results hold up to heavy statistical analysis?

The harm of bad research

Every shoddily constructed experiment and instrument, every poorly interpreted or deliberately manipulated set of results, every anecdote held up as firm “evidence” across the board–all these things do absolutely nothing to further your cause, and in fact do much to harm it.

This is one example of what happens when people push bad research into the general consciousness.  And before you say “Well, bad magical research never killed anybody!”, here’s a sizable collection of recorded instances of people being injured or killed by the misapplication of everything from faith healing to dream interpretation (and, apparently, also GPS systems).

Have the correct tools, and be willing to be wrong

And before anyone tries to start a science vs. magic debate, or argue that there’s no such thing as objective reality*, my point that I am making is that if you are going to claim that magic can be proven through experimentation, then your methodology needs to not be half-assed.

If you are going to claim that you have any authority on anything that involves proving something exists objectively, then you need to be literate in the methods used in proving something exists objectively.

Finally, understanding the basics of research methodology is an incredibly valuable part of critical thinking skills, skills that are woefully under-represented in magic and spirituality, and really are a necessary part of being human.**

Those last three paragraphs that I just wrote right up there? THAT’S the intended take-away. You want to prove magic (or any other similar force or concept) exists in an objective, consistently measurable manner? Then have the correct tools, and be willing to be wrong, if that’s where the evidence and statistics end up taking your research.

* I’m not avoiding them because I don’t think they’re good topics of debate, but I want to keep things focused on the actual topic I’m discussing here, rather than getting derailed. Thank you for respecting that.

** Even people who have never, and will never, run a formal experiment still benefit from knowing the basics of research methodology so that they can have a better idea of what the people who do those experiments tell the general public through their published results (and why that’s important to everyday life). Yes, people who are experts in their field and have access to knowledge and training the rest of us don’t do have an advantage and authority. But knowing the basic processes by which they acquire their knowledge, to include research methodology, can help those of us on the general level of “consumer” of information and products to have a better understanding of why, for example, “studies show Brand X is the best!” or parse out whether a news story on “This food/medication/material COULD KILL YOU” is worth paying attention to.

This article was first published at Therioshamanism.com.

The author

Lupa is an author, artist, and neoshaman living in Portland, OR. She earned her Master’s degree in counseling psychology in 2011, with a specific focus on ecopsychology, as a way to integrate the healing and intermediary work of shamanism with a broader cultural and humanistic framework. When she isn’t engaged in creative chaos in her studio, Lupa may be found hiding out in the Columbia River Gorge. Otherwise, she’s online at http://www.thegreenwolf.com and http://therioshamanism.com.