Robert Slavin, a huge figure in evidence-based education, has written a blog post claiming that John Hattie is wrong. Hattie pursues the approach of meta-meta-analysis. In other words, he brings different meta-analyses together to compute an overall effect size.
Slavin points out that many of the studies that sit underneath these meta-analyses are weak, poorly designed and often don’t seem to relate very well to the concept that is supposedly being investigated. It’s worth mentioning that Hattie accepts at least some of this criticism. That’s why Hattie insists on an effect size above d=0.40 rather than zero. However, Slavin notes that really well-designed studies rarely generate an effect size this large. Hattie is effectively filtering out the good stuff in order to make conclusions based on what is left.
One specific point Slavin makes is about experimenter-designed tests versus standardised tests. In my view, the former are valid and it is reasonable to make use of them in basic research. However, they will be more sensitive to the concept that is being researched than standardised tests and so will give a larger effect size. You cannot just mush together these very different measures.
I think Slavin makes a powerful case, but we could go further. It is not just the design of a study that can affect the effect size, the age of the students will also matter, with younger students typically generating larger effect sizes due, in my view, to their relatively rapid rate of learning in quite a restricted domain of knowledge. So we should always ask the age of the subjects when an effect size is quoted.
I also think that the concepts that these effect sizes are associated with need to be much more tightly defined. There has been a small industry devoted to answering the question: what does Hattie mean by ‘feedback’? The answer is: Lots of sometimes contradictory things.
It is worth noting that Hattie is not the only one engaged in meta-meta-analysis. It is the process behind the Education Endowment Foundation’s (EEF) Toolkit, a Toolkit licensed to Evidence for Learning (E4L) in Australia. Each strand represents a broad, vague concept and draws on evidence from a range of meta-analyses as well as the much higher quality randomised controlled trials conducted by the EEF itself. In other words, it synthesises apples and oranges.
The strand I have investigated the most is ‘meta-cognition and self-regulation’. What even is that? It’s hard to tell. Yet the territory, as usual, is full of contradictions. While the EEF released a report warning against generic thinking skills programmes, sitting at the heart of the randomised controlled trial evidence for this strand is ‘Philosophy for Children’ (P4C), a generic thinking skills programme.
You may think that the fact that P4C underwent a randomised controlled trial means that the data supporting its use is sound. Not so. The study was well designed and generated no effect on the measures specified prior to the trial, a null result. However, once they had the data, researchers reanalysed it and came up with a small effect size.
If true randomised controlled trials can generate misleading effect sizes like this, then what monsters wait under the bed of the meta-meta-analysis conducted by Hattie and the EEF?
Filling the pail 
It gets worse than this, Hattie often does not represent the studies faithfully. I challenge anyone to read the 3 class size meta-analyses Hattie used in his 2009 book and come up with the same conclusion as Hattie has. I recommend starting with Gene Glasses meta-analysis -see here – https://visablelearning.blogspot.com/p/class-size.html
Then if that’s not enough, read the 1-meta-analysis Hattie used to denigrate any of the welfare work schools do-
https://visablelearning.blogspot.com/p/diet.html
Hattie’s polemic -‘disasters’, ‘going backwards’, ‘also-rans’, ‘distractions’, on these and many other aspects (influences?) on achievement needs to be challenged.
Do well-designed studies tend to contradict Hattie though? I see Hattie’s recommendations, and then see good studies that tend to back them up. Teachers that follow his suggested methods seem to me to be better teachers than those that follow others with high reputations, say Boaler.
If his conclusions come to the same conclusions as “better” studies, albeit with smaller numbers, perhaps there are better targets to shoot at.
I would like to see more examples of the type George has above, where instead of complaining about his techniques, people show that Hattie is wrong in his *conclusions*.
(I’ve believed for many years that class size is important — but to teachers, not students. The effect of techniques and ideas on teachers is almost never considered. Yet many techniques are good in theory but impossible for teachers to keep up in practice without burn-out — I’m looking at you, triple marking!)
I’m not sure about this argument at all. Our first duty as teachers is to intellectual honesty. If I go easy on Hattie’s methods because I find some – and certainly not all – of his conclusions convenient, then I am creating fertile ground for a new generation of debunkers who will question Hattie’s methods and conclusions. This can only set back the cause of anything he is right about.
Fair enough.
I do think that targeted criticism is more effective than general though. If he’s wrong then there should be no problem finding *examples* of where he is wrong.
Articles that merely go “he’s wrong” and stick to generalities don’t persuade me very much.
Slavin gives a number of specific examples of why Hattie’s approach of using effect sizes is flawed. I add to this. This is fundamental to whether we should trust Hattie’s conclusions. These are not vague, unspecific criticisms.
Hattie mania is just another example of the education world falling for yet another false holy grail. Apart from the feedback issue, Hattie can’t quantify the traits of teachers who can make his ideas work. And the age issue is critical: rates of apparent progress do drop significantly as competence rises. Young children learn quicker. But I have never come across anyone taking this into consideration when applying Hattie’s theories. And it all still only relates to exam performance, which plenty of evidence suggests (see Robert Bjork) is not necessarily the same as true learning.
Reblogged this on The Echo Chamber.
Pingback: What if John Hattie is right for the wrong reasons? – Filling the pail
It seems to me that Marzano is other major advocate of the meta-analysis in education, but I’m yet to find the same level of scrutiny brought to his use of it, despite his work being increasingly promoted in my neck of the edu-woods. (Hattie, EEF and Marzano dominate the footnotes of current Victorian Education Dept advice, eg: https://www.education.vic.gov.au/Documents/school/teachers/support/highimpactteachstrat.pdf .)
Can anyone point me towards critiques that bring Marzano into this conversation?
try Dr Jonathan Becker – http://edinsanity.com/2009/06/02/marzano_part1/
and
https://www.fitzforschools.com/single-post/2016/08/10/Why-OCPS-Teachers-Leave—Fitzpatrick-Shares-40-Reasons-to-Replace-Marzano
have a look at – http://education.modernsense.net/2012/10/marzanos-database.html
The same problems are presented – misrepresentation, poor research and excessive inference.
Greg can’t help but agree with you on this one. Well done.
Pingback: Are effect sizes magic? – Filling the pail
Perhaps looking at what studies Slavin and Hattie and Marzano and others have in common is a productive way to help busy teachers. If I do “x” in my classroom what are its impacts on my students? I have gotten to be a fan of Project Zero’s Visible Thinking because it promotes in me and my students looking at the impact of work in classrooms.
But researchers differ so much and often contradict each other e.g Marzano says non-linguistic representations is a high strategy whereas Hattie says simulations and visual representations are a very low strategy – I’ve compared some of them here – https://visablelearning.blogspot.com/p/other-researchers.html
Plus Hattie and Marzano have too many conflicts of interest. We need to go to more independent organisations, although Greg and others have already found problems with the EEF.
To me, one source that is independent and reliable is the U.S Education Dept – ‘what works clearing house’ which employs more than 23 Professors and PhD students from a variety of fields. They use a very strict quality of evidence protocol (which neither Hattie nor Marzano do) which results in a more honest appraisal of the evidence out there. What is clear to me from this is that the evidence for anything is not that strong.
Why don’t we trust good teacher’s until the evidence is stronger?
A simple example of trusting Hattie and Marzano more than good teachers is – the Dept of Education (in the same state as Hattie – Victoria Australia) produced it’s 10 High Impact Teaching Strategies (HITs) largely based on Hattie and Marzano. It rolled these out to more than 50,000 teachers in 2017, including a significant teacher accountability process. One of the HITs is ‘worked examples’ based on 1 Phd study that Hattie has used for 15 years.
But Hattie just added a 2nd study with a very low effect size of 0.16 which then brings ‘worked examples’ below his 0.40 hinge point.
So if you rely on this evidence ‘worked examples’ was a HITs in 2017 but not one in 2018!
One of the cornerstones of the scientific method is reliability. Until educational research for particular influences is replicated we should not rely on it.
Pingback: The Promise of Evidence-Based Guidance – The Society of Buried Treasure
Just catching up on this blog and the Slavin, Bergeron crits. Ok, Hattie’s deck of cards has collapsed. What now? A return to gurus and personal anecdotes? Perhaps we can cling to odd piece of research that isn’t torn apart… but trainee and new teachers need guidance. What can we suggest?
Well, there is a book I could recommend…
Pingback: Do my standards of evidence follow my preferences? – Filling the pail
Pingback: Is Maths Pathway evidence-based? – Filling the pail
Pingback: Is scientific evidence ‘phallic’? – Filling the pail
Pingback: Evidence-informed teaching – Filling the pail
An important message for sure; but what really caught my attention was the photo (Filling the Pail)
As we move from ‘Sage on the Stage’ to ‘Guide on the Side’ I hope we are not throwing out the baby with the bathwater (to carry the water analogy further.)
If we don’t fill the pail to some extent (content knowledge) how do we manage the fire we build in students?
Not saying this blog does that at all, as I haven’t given it an in-depth look.
In fact, I am having to use my own ‘evidenced based hack’ on myself right now; I find the picture so compelling, I want to check out the blog in detail, but as I am working on a time sensitive project, this brief foray into the comment section is just the Post It note to myself come check this blog out in more detail…looks interesting.
Love the photo, it goes without saying…
Pingback: Does it all depend on context? – Filling the pail
Pingback: Defeating maths anxiety – Filling the pail
Hello, regarding – “One of the HITs is ‘worked examples’ based on 1 Phd study that Hattie has used for 15 years.”
I’m beginning to review Hattie’s work. Can you provide more details on this study? A link is sufficient. Any more information is also appreciated.
The first was a PhD study giving an effect size of 0.57. Hattie has just added a 2nd meta-analyses with a very low effect size of 0.16. The PhD study is detailed going through much of the theory on worked examples. You can find them on this page – http://visablelearning.blogspot.com/p/worked-examples.html
Thank you.
Pingback: More criticism of Jeffrey Bowers’ phonics paper | Filling the pail