There are many ways to mess things upPosted: December 11, 2015
Image you have an educational programme for teaching literacy or a numeracy intervention. Imagine that it has five key elements, that you’ve trialled the programme extensively in small studies and you now want to scale it up across lots of classrooms. How’s that going to work?
Well, there is only one way that teachers can implement all five elements. However, there are five ways in which they could miss one of the elements; they could miss element 1, 2, 3, 4 or 5. There are 10 ways that they could miss two of them; 1 & 2, 1 & 3 and so on. If you continue with this, you find that there are 31 different ways of not fully implementing the five elements.
However, these different ways are not all equally likely. Imagine that you run some excellent training for staff so that they largely understand what the programme is trying to achieve. They end up with an 80% chance of implementing each element. This means that the chance of implementing all five elements becomes 0.8 x 0.8 x 0.8 x 0.8 x 0.8 or approximately 33%. In other words, your programme will be implemented in full in about a third of classrooms.
I think that there are two clear implications of this. Firstly, as I have argued before, a good programme needs to have a positive effect if only some of the elements are faithfully implemented. If the impact is negative unless implemented in full then we should probably steer well clear.
Secondly, this sheds some light on the scripting of lessons that is a feature of Engelmann-style Direct Instruction programmes. Apparently, Engelmann did not set out to do this but found himself on this path when teachers struggled to implement the programme fully. We can see why this might arise even if teachers have a pretty good understanding of what we are trying to achieve. We can also perhaps imagine how scripting was therefore a key factor in the success of DI in Project Follow Through.
I can’t help comparing this to Atul Gawande’s checklist approach. Yes, we can expect doctors to understand the importance of washing their hands but if we put this on a checklist and insist on use of the checklist then the chance of this happening consistently will be much higher.