Peer review of my article for Impact magazine

Yesterday, I published an article that had been rejected for publication in Impact magazine, the journal of England’s Chartered College of Teaching.

Following publication, and a discussion about this on Twitter, the College made claims about the contents of the review comments. Specifically, they claimed that, at no point did reviewers take issue with the opinions in my piece:

I have previously asked the College for permission to publish the review comments and they declined on the basis that they had not sought permission from the reviewers for me to do this. I therefore asked the College if they would seek permission from the reviewers. They replied that this was not something that they would do.

The Chartered College has been set-up with five million pounds of UK taxpayer money after failing to raise sufficient cash through a crowd-funding campaign. Many UK teachers have concerns about the College, how it will compare to the defunct and disliked General Teaching Council and whether it will become a mechanism for educationalists to impose their ideas on classroom teachers. In addition, the article that I wrote is critical of the approach of the Educational Endowment Foundation (EEF), another body in receipt of large amounts of public money. That is why it is in the public interest to scrutinise the College and the EEF, and that is why I have published details of the process of submitting my article, something I would not do if submitting an article to an academic journal.

Given the public interest argument, the fact that the reviewers are anonymous and cannot be identified in the reviews and the fact that the College have now chosen to make claims about the content of these reviews, I have decided to take the step of publishing them in the link at the end of this post. Paraphrasing them was never going to be satisfactory because of the potential for unconscious bias in how I completed this task.

There are a few points to note. I sent a first draft to Impact and received the first three reviews. I then revised my piece on the basis of the first and second review in order to make it more clear that I was criticising both the use of meta-analysis in general and the way the EEF used meta-analysis. I also fixed some relatively minor points, including one about a missing reference. It is the revised piece that I published yesterday. As requested by the College, I provided them with a detailed, point-by-point response to all of the issues raised in the first three reviews. These were mostly points raised by the first reviewer because they were far more numerous.

The College then sent out my revised piece to four new reviewers.

I will not comment on the reviews in this post because I would like you to make up your own mind. However, there is one potential point of confusion that is worth clearing up. One reviewer wrote that, “the assertion that the Toolkit indicates that a teacher will get 8 months’ progress is wrong and misleading.” If you look at the toolkit today, you will see that the figure is +7 months of additional progress and so you may therefore think that the reviewer’s point is that I got this figure wrong. However, this figure was altered on the 13th April 2018. At the time that I wrote the article, and at the time it was reviewed, this figure was +8 months.

You can read the reviews here.


14 thoughts on “Peer review of my article for Impact magazine

  1. What I take from those reviews more than anything else is that they’d pretty much all made up their minds about your piece before they’d read it.

  2. Seems to me that your article was well written for a magazine rather an academic journal. You want some journalistic flair and flow to grab your audience in a magazine. The reviewers seem to be holding out for the style of writing appropriate for an academic journal. However, it is hard not to form the impression that their comments regarding tone and balance disguise an unease with the points being made or at least a philosophical opposition to them. It doesn’t appear that lively informed debate and criticism is going to be part of the Impact Magazine’s oeuvre. How about submitting it to the new researched magazine?

    • Chester Draws says:

      If Impact is going to be treated as an academic journal stylistically, then it will have almost no actual impact. Teachers simply are not going to wade through verbiage to get to the actual point, which has to be hidden so as to not offend anyone who disagrees.

      If Impact is to be read by teachers, then getting rid of “academic style” is a must.

      Magazine articles are not, of themselves, less rigorous than academic ones. Academic writing is one of the curses of modern learning.

      • If teachers wish to read research I am afraid quite often they are going to have to wade through academic verbiage – and work out what it means. There is little research, as far as I can tell, that makes itself available to teachers in a readable form. Greg is a breath of fresh air in highlighting and discussing research.

  3. Chester Draws says:

    It seems to me that what they want is not an opinion at all. What they want is the (usual) academic re-hashing of previous findings and then some sort of synthesis of that, perhaps with your own opinion buried towards the end. That way no-one gets left out.

    So you will have to do a literature review if you want to get past these reviewers, done in “dispassionate” tone.

    Of course generally only an academic has the time to complete a full literature review. If they expect that, then any non-academics are basically excluded from the get go.

    Then, of course, the average reader declines to read three pages of turgid recapitulation of research to get to the meat of an article.

    Contrary opinions aren’t banned outright, you just enter a Kafkaesque set of arcane rituals to get to where you need to get to in order to be allowed to say them. And academics wonder why people are so scathing of them!

    But a well written blog article will reach who you need to better than any academic one anyway.

  4. Tom Burkard says:

    ‘Peer review’ as originally conceived by Ibn al-Haytham in the 10th century was an enormous advance in scientific thinking–it is essential to the process of falsification. However, as conceived by all too many modern scholars, it is used to bolster orthodoxy and discredit attempts to falsify it. This is especially true in education, which has come very late to any notions that the discipline could be treated as a science.

    I’d be interested in seeing peer review of Greg’s article by cognitive scientists.

  5. Michael Pye says:

    The reviews were interesting. i agree with education86466 that you have written in a more journalistic manner and impact has a different st;e of writing so I have some sympathy with the tone and style argument. However Impacts style isn’t actual very good, reminding me of most action research pieces than more thorough analysis, and definitely not of the most eloquent and engaging writers such as Plinker or Williams who are quite prepared to admit an opinion. In Gregs favor is that fact that unlike most of the impact articles he attempts to predict and refute the likely counterarguments which seems to be a unusual. Ironically impacts style reads as very impartial and factual while being in fact extremely opinionated. While I though the first reviewer was trying to be balanced I did find the comments around “Does the author demonstrate a thorough knowledge and understanding of the
    research?” interesting. No explanation of the missing research was given and the normal style of the journal seems to be rather incestuous referencing predominately supporting ideas and research and refusing, seemingly on principle to identify, analysis and critique genuine controversies. In summary i can see why they rejected your piece though I am not sure that they are in fact making better or even fairer arguments.

  6. Point of factual accuracy. Reviewer 4’s (post revision) statement: “references to statistical significance which are of little relevance in cases where effect sizes are reported with confidence intervals” is wrong. There were no confidence intervals around the effect sizes in the P4C trial, nor in the EEF meta-cognition and self regulation review. In the technical appendix the range of the different effect sizes is reported, which might have confused reviewer 4, but this is quite different from the confidence interval around the effect size. Reviewer 4 is correct that confidence intervals would have negated the need for tests of statistical significance. The problem is they weren’t reported. The very point that Greg was making.

  7. Pingback: Accuracy, opinions and tone – Filling the pail

  8. Pingback: A new report on ‘metacognition and self-regulated learning’: What does it mean? – Filling the pail

  9. Philip Beadle says:

    I thought what you have written is important, but you are challenging the methodologies of the establishment. There are factual errors in the reviews: P4C is not genuinely metacognitive and self regulation is a subset of metacognitive strategy.

  10. Pingback: Learning lessons from England’s Chartered College – Filling the pail

  11. Pingback: Peer Review | Filling the pail

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.