I wrote an article for Impact, the magazine of England’s new Chartered College of Teaching. They declined to print it. They had sent it out to two sets of reviewers who took issue with a number of aspects of what I had written. Given that it was essentially an opinion piece, I expected comments on accuracy: Were my figures correct? Had I correctly explained statistical significance? And so on.
I was surprised to read comments objecting to my tone or simply disagreeing with my opinions. Tone is largely subjective. Anyone reading my piece who is personally invested in metacognition and self-regulation will find the tone confronting. Others will simply find it playful.
When I commented on Twitter about the fact that the reviews took issue with some of my opinions, the College Twitter account said that I was wrong:
This is what forced my hand and made me decide to release the anonymous reviews.
Following this, some defenders of the College engaged in a Pythonesque Twitter thread where they attempted to deny the obvious. As far as I understand it, they claim that the reviews did not take issue with my opinions because they gave good reasons for taking issue with my opinions. Or something. For me, this peaked with the following exchange:
However, what has perhaps been lost in this process is the notion of accuracy. When I was sent the final set of reviews, I was given a couple of days to revise my article for tone and accuracy. I declined, partly due to the fact that I didn’t see the need and partly because of the time I was given.
One reason I didn’t see the need is that there were so few comments relating to accuracy. Indeed, one reviewer seemed to indicate that s/he lacked the knowledge to comment on accuracy and instead suggested my tone was ‘Clarksonesque’. Those who did made comments about accuracy appeared to be wrong.
For instance, one reviewer claimed, “the assertion that the Toolkit indicates that a teacher will get 8 months’ progress is wrong and misleading.” On the 13th April, the Toolkit revised this figure down from 8 months to 7 months, but at the time of writing, and at the time the review was written, it was indeed 8 months. So what was wrong with my claim? It is obviously a nonsense that anyone could expect to get 8 months progress from implementing one of these diverse approaches but this is what the Toolkit suggests and this is one of the reasons I am critical of it.
One reviewer noted that my opinion contradicted that of the Education Endowment Foundation (EEF) but didn’t really resolve the issue, despite seemingly showing sympathy for the EEF’s position. The comment seemed more of a caution to the editor than something I could respond to.
There is one important claim about accuracy in the reviews that could be a valid point. After criticising my ‘selective’ use of a Dylan Wiliam quote, a usage that Dylan Wiliam did not object to in his own review, Reviewer four mentions, “…references to statistical significance which are of little relevance in cases where effect sizes are reported with confidence intervals.”
Jim Thornton, professor of obstetrics and gynaecology at Nottingham university and an expert in randomised controlled trials, took up this point in a comment on an earlier post:
“There were no confidence intervals around the effect sizes in the P4C trial, nor in the EEF meta-cognition and self regulation review. In the technical appendix the range of the different effect sizes is reported, which might have confused reviewer 4, but this is quite different from the confidence interval around the effect size. Reviewer 4 is correct that confidence intervals would have negated the need for tests of statistical significance. The problem is they weren’t reported. The very point that Greg was making.”
So I’m not really sure how I could have revised the article for accuracy.
Notes: You can find my original submission, prior to any amendments I made on the basis of the first three reviews, here. You can find my response to the first three reviews here. The sixth response should read ‘The fact that teaching methods..” but I’ve left it as submitted for transparency.