A new report on ‘metacognition and self-regulated learning’: What does it mean?

Embed from Getty Images

In 2013, the Education Endowment Foundation in England (EEF) ran a randomised controlled trial of Philosophy for Children (P4C), a programme aimed at improving children’s thinking. A typical activity might involve a discussion about whether it is wrong to hit a teddy bear.

Before the trial, the researchers published a protocol which stated the measures they would examine in order to determine whether the trial had been a success. This is good practice for two reasons. Firstly, we know the trial is taking place. A lot of educational trials are not published if they find no effect and this leads to publication bias. However, preregistration means that future researchers will know about these trials and may still take them into account. Secondly, by specifying the measures that they will use in advance, researchers avoid the problem that is commonly – and ironically, in this case – known as ‘p-hacking’. This is the act of slicing and dicing the data once it’s been collected until you find something that supports your conclusions, then focusing on that measure in your final report.

As might be expected, P4C lessons had no impact on the academic measures that had been preregistered by the researchers. How could it? How could discussions of teddy bears improve maths performance? So you might think that was then end of this rather pointless story.

But no. Once the data was in, the evaluators sliced and diced it anyway and found a new measure which they claimed showed an impact. By chance, the control and experimental groups had different mean performances in reading, writing and maths at entry, with the P4C group having the lower averages. This gap narrowed after the trial, although not by much. It was then suggested that this was due to the P4C intervention and a whole lot of media coverage ensued. Others have suggested that it is an example of a well-known statistical artefact known as ‘regression to the mean‘, although the lead evaluator says they checked for this.

The evaluators did not report the standard test of statistical significance, a p-value, for this measure because they object to such tests. Given that the research involved public money endowed by British taxpayers, I don’t think this was their decision to make. Nevertheless, even a statistically significant result on a measure derived from torturing the data should be looked at with deep skepticism.

So perhaps we have now finally reached the end of the tale? No. The confirmation bias is strong with this one.

For a number of years, Kevan Collins and Jonathan Sharples of the EEF have been promoting the benefits of ‘metacognition and self-regulation’ (e.g here and here). The EEF have produced a toolkit; a set of reports intended to help schools choose effective interventions which it groups into ‘strands’. ‘Metacognition and self-regulation’ is one such strand and I wrote a critique of this strand for an issue of Impact magazine, the trade journal of England’s Chartered College of Teaching, that they declined to print (you can read it here and the subsequent discussions here and here).

P4C is one of the studies listed under this strand. Despite the weak 2013 results, the EEF have gone ahead with a scale-up study, costing £1.2 million which is roughly 1% of all the money endowed to them by UK taxpayers. Presumably, this will also demonstrate no effect of the P4C lessons on academic performance.

Yesterday, the EEF released a guidance report on metacognition and self-regulation which makes interesting reading. The report is sensible enough and draws evidence from studies investigating reading comprehension strategies, the use of spaced practice, teaching children how to plan and edit their writing and teachers modelling their own thinking, a key component of explicit instruction.

My main issue with the report is that I don’t think all of these approaches should be grouped under the same heading because they are too diverse. At the very least, I would argue that evidence for self-directed learning should be placed in a separate category to evidence about the effectiveness of explicit instruction because, although often used in complementary ways, they represent fundamentally different forms of learning. This is not just my view, researchers have identified key conceptual difficulties in reconciling the research evidence in these different areas.

However, I must commend the authors on stressing the importance of relevant knowledge in the use of metacognitive strategies. For instance, it was good to see this caution about writing an essay on a Shakespeare play:

“We cannot adequately deploy metacognitive strategies for monitoring and evaluating our essay-writing if we do not first understand the components of a successful essay and have a knowledge of Shakespeare’s world.”

The authors expand on this point:

“There is little evidence of the benefit of teaching metacognitive approaches in ‘learning to learn’ or ‘thinking skills’ sessions. Pupils find it hard to transfer these generic tips to specific tasks.”

In other words, you can’t teach improved thinking in some kind of general way, as Dan Willingham has been arguing for some time.

What is notable about the report is that it does not mention or reference P4C or the cognitive acceleration programmes that have formed such a central plank of the EEF’s own research into metacognition and self-regulation. Why?

Well, for one thing, P4C looks very much like a discrete thinking skills programme. Cognitive acceleration, as tested by the EEF, is a science intervention, but it is also intended to produce a general improvement in thinking ability that transfers to other subjects. So both would seem to fall foul of the advice in this new report.

What should we make of this? One the one hand, the EEF commissions £1.2 million worth of research into P4C on the basis of a questionable pilot trial result and the notion that it is an example of a metacognition and self-regulation strategy that is worth pursuing. On the other hand they publish a report into metacognition and self-regulation that does not even mention P4C and, if anything, would seem to caution against such an approach.

If you would like some examples of metacognitive strategies then you can follow the links here and here 😉

Advertisements

10 thoughts on “A new report on ‘metacognition and self-regulated learning’: What does it mean?

  1. HI Greg

    I’m a longtime reader, first time poster. Usually I’m a big fan of your writing, but I think you’re a bit wide of the mark here.

    You make lots of important points, but I think there are two things worth reconsidering.

    First, in terms of the trial, you imply that EEF funded P4C simply because they think it is great (perhaps they do!). However, would it not also be legitimate to evaluate things that are popular with schools, but where the evidence is unclear? I think it’s hard to draw strong inferences about P4C from the first trial, so isn’t it worth a replication study?

    Second, I agree that metacognition is a broad term. However, isn’t the point of the report to be useful to teachers? I think it does a great job of that even if it isn’t as intellectually pure as you might want it to be. There’s a trade-off between accuracy, accessibility and usefulness. How would you organise the structure differently?

    Keep up the interesting posts.

    1. Thanks for the kinds words

      1. I have no problem with P4C being popular in schools, just as I have no problem with story time or PE. Done well, P4C could help grow children’s knowledge of the world. However, I think it is eccentric to propose that P4C might have a measurable impact on standardised English and Maths scores. How exactly is this supposed to happen? Given that there really *was* no measurable impact in the pilot trial on the measures that the researchers had set-out prior to the trial, I really do not think it is justified to spend over a million quid running a larger trial.

      2. The report may well be useful to teachers. I have gone to the trouble of highlighting what I believe to be some of its strengths. However, I don’t think categorising such diverse approaches together in this way is justified. Why should I not say so?

  2. Greg,
    EEF and others know what they are producing is useless but in order to keep the money flowing into wasteful RCTs in education they attempt to fabricate meaning from the detritus.
    The BBC have contributed to metacognition in a much less expensive manner, one that has found appreciation and support well beyond a confused teaching profession.
    It can be found here.

    1. Pretty sure they are doing it because they think it works. Bias is more likely then deciet. Also cunk on Britain is hilarious but the link to metacognition is a stretch. Most of Brooker’s gags are malaprops and reversals of common saying ideas and sayings.

  3. Greg,
    I note with interest the contributions of those who are quick to critique and even offer advise on your polite criticisms of EEF and their wasteful RCTs.
    I wonder why they have to wait for someone else to do the work of constructively unpicking EEF activity and then presume that their contribution balances out the equation? I learned a long time ago that education research is an oxymoron. Those anxious to participate do so largely for selfish career and financial interest and their ignoring of Ronald Fisher’s instructions on the design of experiments proves the charge.
    If there wasn’t the availability of shrink wrapped statistical software programs for data-mining – teachers, psychologists and education researchers wouldn’t contribute so much incoherent nonsense posing as science.

  4. “… a simultaneous equation.”

    The quote is
    “2. Monitoring:
    “Has this improved my
    understanding of the task?
    Yes, it now looks like a
    type of problem I’m
    familiar with: a
    simultaneous equation.”

    And this is the first example !!!!!!!!

  5. The EEF enterprise (along with that of John Hattie and others) of deciding if an educational area is more or less effective on the basis of blending together effect sizes from different studies has been shown to be a huge mistake. It’s not just they combine studies that don’t seem to be of the same type, it’s that effect size isn’t a measure of how effective an intervention is anyway. P4C appears to have been recommissioned because they were impressed by the effect size, but this is not a measure of how effective the intervention is anyway. See https://bit.ly/2oyEWnn or listen to a recent podcast on the issue (https://bit.ly/2w8eyXx)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.