I was alerted* to a new paper by Hans Luyten, Christine Merrell and Peter Tymms about effect sizes. Effect sizes get bandied around a lot in education and many will have heard of John Hattie’s figure of d=.40 (0.4 of a standard deviation) as an effect size worth having. However, this is pretty blunt. We know, for instance, that effect sizes may be impacted by a number of factors, including the age of children.
Luyten and colleagues decided to try measuring some baseline effect sizes. In other words, they decided to determine the effect for normal teaching in different subject areas and for different groups of students. The students they chose were primary students in England who primarily attended independent schools and effect sizes were calculated for reading, mental arithmetic, general mathematics and a construct called ‘developed ability’ that included picture vocabulary and pattern recognition. They used a regression-discontinuity approach where very similar students in two different year groups are compared i.e. children falling either side of an arbitrary cut-off date for entry into a particular year group.
The good news is that schooling seems to have an effect. However, the effect sizes varied. They generally became smaller as children grew older. For reading, the effect size shrank from d=.55 when comparing Year 1 and Year 2 to d=.08 when comparing Years 5 and 6. Is this an indication that, once children can decode, reading cannot really be taught? See this Hirsch article for a discussion relevant to this idea.
It is therefore tempting to suggest that, if this is representative of students more generally, a reading intervention in Year 5 with an effect size of d=.30 would be worth having. However, we tend to intervene with struggling students who may in fact look more like the younger students in this study. So what effect size should we be seeking?
I think that it is impossible to say.
The changes in effect size for general maths are less severe but still significant, ranging from d=.47 to d=.27. And this only goes as far ad the end of primary school. What would these look like at secondary?
The evidence is now building. We need to move away from tables of effects such as those produced by Hattie and the Education Endowment Foundation toolkit. Effect sizes will depend on student age and level of advancement in their learning. Claiming that amorphous intervention X will deliver 4 months extra progress is invalid and unhelpful. Instead of comparing effect sizes and hoping they cross an arbitrary threshold, we need to do more studies that compare Intervention A with Intervention B and a control. In the long term, this is the only way we will remove the junk from our data; by comparing like with like.
*Best Evidence in Brief is an excellent email sent out by the Institute for Effective Education in York, England. It’s well worth subscribing to.