Is ‘metacognition and self-regulation’ an actual thing?

When I google the term ‘metacognition and self-regulation’ (in quotes), the top hits are from the UK’s Education Endowment Foundation (EEF) and Evidence for Learning, an organisation that uses and promotes EEF materials in Australia.

I’ve long argued that this term is a problem because it seems to contain a bunch of things that are largely unrelated to each other. My interest was piqued when I spoke to someone from the EEF who was quite critical of Nick Gibb, England’s school minister, and Gibb’s focus on phonics and subject knowledge. Instead, EEF evidence was apparently pointing in the direction of ‘metacognition and self-regulation’.

The EEF use ‘effect size’ as their main measure of an educational intervention; something they then translate into additional months of progress that the intervention will supposedly provide. This is flawed because it’s not appropriate to compare effect sizes for studies with different types of outcome test (experimenter designed versus standardised), different subject matter, different age groups, different cohort types (selective or full ability range) and different study designs.

You see this effect in the EEF results; its own randomised controlled trials (RCTs) often result in lower effect sizes than studies using weaker methods. Nevertheless, it’s worth looking at the data that goes into the EEF’s measure of metacognition and self-regulation. I’ve summarised the studies, plus one additional study, in the table below:

The studies in italics are ones that the EEF have drawn from a review of the available literature. They typically consist of meta-analyses of a range of studies that are not all RCTs. I have stated the kind of outcome that was assessed, as best I understand it, in brackets.

Below these are the studies carried out by the EEF. The final EEF study listed is for, ‘Let’s Think Secondary Science’, an intervention based upon the longstanding Cognitive Acceleration in Science Education (CASE) materials. The EEF have not yet included this result in their overall weighted average for metacognition and self-regulation, but they should. ‘Let’s Think,’ is actually very similar to the, ‘Thinking Doing Learning Science’ intervention for primary students which is the subject of the other Hanley et al. study in the table.

For completeness, it’s worth sharing that the two Gorard et al. references reflect the two different outcomes of the (in)famous Philosophy for Children trial and the NIESR references reflect a study involving mindset. Torgerson et al. evaluated an explicit writing intervention.

Given this information, how should we interpret the EEF’s claim that, overall, implementation of a metacognition and self-regulation strategy will result in an additional 8 month’s progress and is, “High impact for very low cost, based on extensive evidence’?

Firstly, apart from the explicit writing intervention evaluated by Torgerson et al., school leaders should probably avoid the rest of the EEF interventions. Even the one other clearly statistically significant result, with an effect size 0.22, maps to far less than 8 additional months progress (I make it about 3 months according to the EEF conversion chart).

We may achieve a larger effect if we implement one of the non-EEF interventions. However, it was the problems with the methods used by these kinds of studies that prompted the foundation of the EEF in the first place.

The second, and perhaps most fundamental, point that the table highlights is the disparate nature of the trials and outcomes. What does an explicit writing intervention have in common with a mindset intervention or philosophy for children?

Explicit attempts to teach reading and writing strategies have a long track record and are clearly defined, even if the former may provide a limited, one-off benefit. Would it not aid clarity to give these interventions their own categories? After all, we should not assume that adopting a mindset intervention will lead to the same effect as an explicit writing intervention.

Not only are these programmes qualitatively different to the others in the metacognition and self-regulation basket, the mechanism seems far more obvious. It is entirely plausible that teaching children to plan their writing will improve the structure of their writing and lead to higher writing scores. But how is Philosophy for Children supposed to improve maths?

The EEF seem strangely attached to evaluating interventions with obscure or mysterious mechanisms. For instance, in evaluating a core knowledge programme, they taught children about one domain of knowledge and then gave them a reading test in a different domain. Not even enthusiasts for core knowledge would argue this should work because the whole idea is that reading is enhanced by knowledge of the domain, not by knowledge of a different domain.

I wonder whether researchers at the EEF subscribe to a form of genericism, where academic performance is viewed as the result of a set of general, trainable skills. That might explain the way they design some of their studies, as well as their attachment to the metacognition and self-regulation chimera.

For their part, school leaders need to be aware that this category is deeply misleading, that it isn’t an actual thing and that they need to look under the hood if they want to find anything worth examining.