The Education Endowment Foundation (EEF) have released a report into a three-year randomised controlled trial. Schools were allocated to a control or intervention group and the intervention teachers were provided with the a mentor, time off timetable for training, a number of books to read (I’m sure the authors were pleased about this) and, crucially, a video camera and microphone to record and then review their lessons. The control group received none of these things and carried on as usual.
The result showed that, on a standardised assessment, the intervention group slightly outperformed the control in English (mean of 13.76 vs 13.16, p=0.05, d=0.15) and science (mean of 26.67 vs 26.29, p=0.04, d=0.12) but not in maths where the result was not statistically significant. Note that the English and Science effect sizes were very small.
The most likely explanation for these results is an expectation effect such as the placebo effect. The teachers in the intervention expected it to have an impact and undoubtedly reflected more on their classroom teaching than they otherwise would have done – imagine viewing all those videos. They probably planned a little more rigorously and were given the time and space to bounce ideas off colleagues.
The students in the intervention would also have been aware that something was a little different, particularly given the dirty great video camera at the back of the room. And this might have affected their expectations and behaviours.
Funnily enough, this is not the inference that we are invited to draw. The books and training were all based on something called ‘dialogic teaching’ where teachers ask ‘higher order’ questions, based on the work of Vygotsky and Bruner and all that kind of thing.
So we are meant to conclude that it is the dialogue teaching that is effective. This sits well with the narrative that the EEF have been building around ‘meta-cognition and self-regulation‘ strategies and has already been taken up in breathless news reports (hat tip to @BarryGarelick) as a means to bash traditional forms of teaching.
To me, this demonstrates that the EEF is badly in need of reform. They may have already jumped the shark with Philosophy for Children but there is still a chance to get things back on track. We can’t really eliminate expectation effects in studies like this, but the use of ABC designs, where one intervention is pitted against another, would be a step in the right direction.
Correction: The result for English is not technically statistically significant because it has the value of p=0.051 to three decimal places.
Filling the pail 
Reblogged this on Barry_Garelick and commented:
Greg Ashman takes an Extreme Close-Up look at EEF’s report on “dialogic” teaching and its implications.
Reblogged this on traditional math and commented:
Greg Ashman take an Extreme Close-Up shot of EEF’s study on dialogic teaching and concludes as I have that it’s one more strawman used to bash traditional teaching.
I’m very glad you’ve pointed this out Greg. It always makes me want to scream when “evidence” like this is presented to teachers by the usual gang of on-the-make education academics/bureaucrats at staff mettings or inservices. In the unlikely event that they are even challenged as to the validity of such “evidence”, they will shut down any discussion by muttering curtly that it was a “peer-reviewed study”, “proper scientific study”, “proper randomised study”, etc. When examined closely, of course, such studies fall at the very first hurdle of validity due to the reasons you’ve outlined above (there was a particularly egregious case with a study conducted at my own school about five years ago, which is no doubt now being trumpeted by the theorist in question as a splendid vindication of her particular hobby-horse).
All of which suggests that teachers should be somewhat better informed about what constitutes a proper RCT…and what doesn’t (about 95% of such much-hyped studies in the education field falling into the latter category, in my experience).
I know that there are a number of lesson observation tools out there and I’m hoping that Andy Goff might be able to tell you little more about the new product that is coming which we have experimented with and been very pleased with.
There really is no need for a dirty big camera as you say. Just a very knowledge the lessons being observed changes it dramatically for both the teacher and student, let alone The physical presence of a camera.
Pingback: Classroom Talk, Some Thoughts on the EEF Report… | Trivium21c
P.20 of the report
““In order to assess the pedagogic impact of the intervention, lessons were video recorded in a sample of both the intervention and control groups. Lessons were recorded twice, so as to assess development and progress over time. Video recordings of a sample of English, maths, and science lessons were made (1) at the beginning of the trial (week beginning 21 September 2015) to provide a baseline and (2) towards the end of the trial (fortnight beginning 22 February 2016). Fifteen teachers from the intervention group and 11 from the control group agreed to be video-recorded. The intervention group teachers were self-selected in response to our request for volunteers at the July 2015 induction session. The control group teachers were then selected on the basis of school-to-school matching. Each teacher was recorded twice—in phase 1 and again in phase 2 — yielding a theoretical total of 156 lessons (two English, two maths, and two science in each case). In fact, because not all of the designated teachers taught science, the total number of lessons recorded was 134 (67 in each phase). The resulting recordings were subjected to both quantitative and qualitative analysis (quantitative analysis only reported in this report). “
Yes. Some lessons were recorded from both control and intervention groups to monitor implementation fidelity. However, I read this as being a separate process from the one where teachers specifically in the intervention group videoed and audio recorded lessons more frequently to self-reflect and discuss with their mentors.
Hello Greg. Please send a link to the Extreme Close Up look at the EEF report. It sounds very informative
Pingback: Googling dialogic teaching – Steven Berryman