There’s a statistics website that enables anyone to do their own punk research: estimationstats.com. It’s good enough quality for the output to be used in published scientific papers like mine, but it is simple enough for anyone with a basic command of a spreadsheet like excel to conduct their own experiment. Here’s a worked example.
Firstly, we need to do an experiment that is small and self-contained. Perhaps you want to find the effect of drawing diagrams versus making notes on students’ learning. Next, you need to ensure that you gain any relevant ethics approvals. If you are conducting this research through a university then you would need to follow their ethics process (which can be quite laborious). Otherwise, check with your school.
The next step is to randomly allocate students to one of the two conditions. A rudimentary way to do this is to go down your register and alternate the conditions alphabetically. A more sophisticated method uses a random number generator. It doesn’t really matter. Researchers often use a quasi-experimental design where students are not randomly assigned. Instead, the teacher of Class A does something different to the teacher of Class B and so on. I would avoid such a design if you can. In this case, there are multiple confounds and the statistics we are going to use assume random allocation. So I would suggest doing something where different students in the same class form the two conditions.
In the example we are going to use, some will draw diagrams as they listen to teacher explanations and others will take notes. Make sure you label your conditions in such a way that you will remember what they are. In this case, we will use ‘D’ from drawing diagrams and ‘N’ for taking notes. This is preferable to ‘Condition 1’ and ‘Condition 2’ because the latter will cause you to try and remember which one is which, every time you return to the data.
Finally, you will need to test students. Make sure your test assesses the things that were taught during the experiment, as clearly and as objectively as possible. Avoid trying to add in questions that stray outside of these confines. You are unlikely to get any kind of effect with focused questions so adding irrelevant questions will not help. You may wish to deliberately add transfer questions that assess the same deep structure in a different context, but I would advise making it easy to separate the score on these questions from the score on questions that are more like the learning materials. That way, if there is no effect on transfer, which is likely, you can still look at the non-transfer questions alone.
So imagine that we do the experiment and get the following data (I have made this data up for the purpose of this example):
Now it is time to use estimationstats. First, select “Two Groups”:
You need to enter your data into the online spreadsheet. This comes populated with some mock data and the headings “Control” and “Test”:
You need to rewrite the headings, delete all of the mock data (make sure of this) and paste your data across. This is done with a Ctrl-C and Ctrl-V – I can’t get it to work by right-clicking on my mouse.
I recommend selecting a “Cohen’s d” effect size because this is the effect size that is most commonly understood in education:
The only other alteration I would make to the default settings is to label the y-axis with something a little more meaningful than “value”:
Then click “Analyse” and you should get something like this:
Beneath this, you will see a value for the effect size which, in this case, is an extraordinary d=-1.39. In other words, the mean of “Taking Notes” is 1.39 standard deviations lower than the mean of “Drawing Diagrams”. This is statistically significant at the p<.05 level and we can see this visually because the horizontal line that represents the mean of “Drawing Diagrams” does not overlap with the thick vertical line that extends upwards and downwards from the mean of “Taking Notes”. That vertical line represents the 95% confidence interval around the “Taking Notes” mean.
Estimationstats also gives a p-value, but it’s an unusual ‘non-parametric’ one. We won’t go into that here, but the estimationstats website provides a link if you want to read more about it.
If you do your own experiment, you are unlikely to find anything as clear-cut as this, but if you do then you should probably let us all know.
Right. Over to you.