It can be hard to cut through education research because there is such a great volume of it, most of which is not very helpful. So it’s useful to have some heuristics to fall back on. I have a few that I can recommend.
Firstly, I would ignore anything excessively jargon laden or that mentions French philosophers in its abstract. Such papers are unlikely to offer much to a practising teacher. I’ve picked my way through a number of them now and if there is a point to them, it tends to be quite trivial.
If the paper involves an experiment then take a look at the methods section. Surprisingly often this will have a great big hole in it. For instance, the control and experimental groups might be different.
However, I’m starting to think that there is something even more important to look for. Any intervention should have a plausible mechanism. The writers of the paper need to be able to give a good account of how their intervention works. This is important for evaluating the results of any statistical tests because it relates to the ‘baseline’ probability: something that is never measured by the experiment itself.
Imagine, for instance, we randomised students into two groups and got a wizard to cast a spell on one of the groups before we gave both groups a test. Our null hypothesis would be that casting spells makes no difference to the test results. Imagine we then analyse the data and there is a difference. We do a statistical test and find that there is only a 1 in 20 chance of obtaining this difference if the spell had no effect. How would you interpret this result? I’d put it down to chance because I can see no plausible mechanism by which spells can affect test results.
One way you can roughly evaluate a proposed mechanism is to ask: How far removed is the intervention from the desired result? Is it proximal (close) or distal (far away)? It is much easier to understand the mechanism of a proximal intervention than a distal one. A distal intervention is likely to rely on a chain of influences, none of which correlate 100%, so by the time you pass through a few of the links in the chain, any effect may have washed out.
This distinction is made by Castles and McArthur in a fascinating new Nature paper about reading interventions. I met one of the authors, Anne Castles, at the recent Language, Literacy and Learning conference in Perth.
Castles and McArthur suggest that proximal reading interventions such as phonics and vocabulary training have a much better evidence base than distal interventions such as fish oil, coloured lenses or, heaven forbid, chiropractic.
I wish that the Education Endowment Foundation in England would pay more attention to mechanisms. If so, they might pause before throwing even more money at Philosophy for Children, a distal intervention that is intended to improve English and maths.