There are many ways to mess things up

Image you have an educational programme for teaching literacy or a numeracy intervention. Imagine that it has five key elements, that you’ve trialled the programme extensively in small studies and you now want to scale it up across lots of classrooms. How’s that going to work?

Well, there is only one way that teachers can implement all five elements. However, there are five ways in which they could miss one of the elements; they could miss element 1, 2, 3, 4 or 5. There are 10 ways that they could miss two of them; 1 & 2, 1 & 3 and so on. If you continue with this, you find that there are 31 different ways of not fully implementing the five elements.

However, these different ways are not all equally likely. Imagine that you run some excellent training for staff so that they largely understand what the programme is trying to achieve. They end up with an 80% chance of implementing each element. This means that the chance of implementing all five elements becomes 0.8 x 0.8 x 0.8 x 0.8 x 0.8 or approximately 33%. In other words, your programme will be implemented in full in about a third of classrooms.

I think that there are two clear implications of this. Firstly, as I have argued before, a good programme needs to have a positive effect if only some of the elements are faithfully implemented. If the impact is negative unless implemented in full then we should probably steer well clear.

Secondly, this sheds some light on the scripting of lessons that is a feature of Engelmann-style Direct Instruction programmes. Apparently, Engelmann did not set out to do this but found himself on this path when teachers struggled to implement the programme fully. We can see why this might arise even if teachers have a pretty good understanding of what we are trying to achieve. We can also perhaps imagine how scripting was therefore a key factor in the success of DI in Project Follow Through.

I can’t help comparing this to Atul Gawande’s checklist approach. Yes, we can expect doctors to understand the importance of washing their hands but if we put this on a checklist and insist on use of the checklist then the chance of this happening consistently will be much higher.


6 Comments on “There are many ways to mess things up”

  1. julietgreen says:

    Events within the NHS recently coming to light have demonstrated just how important protocols are in certain things. Again.

  2. Totally agree Greg. Both Zig Engelmann and Atul Gawande were part of this week’s topic in my lecture in my year 3 class. Project Follow Through too:-)

  3. Tunya Audain says:

    Checklists — Why Don’t Teachers Use Them?

    Atul Gawande, in his book — The Checklist Manifesto — mentions several professions that use checklists as both proven standard practice and to avoid ineptitude (and its repercussions). He mentions doctors, lawyers, professors and engineers. Of course we’re familiar with aviation using checklists which Gawande references.

    Why don’t teachers use checklists? Especially teachers of Reading? Most people and certainly most parents see Reading as a very important subject for literacy, for knowledge, for problem solving and for a host of other pedagogical reasons. BUT, the KEY reason why Reading is important is that this skill, gained and fluent, allows the student to get on the ramp of disciplined learning itself.

    Instead, resisting (defying?) proven practices, teachers cling to fallacious beliefs. Note, this term is no redundancy — beliefs can be truths or myths, but it’s when they are false that “fallacious beliefs” can do damage.

    In Preventing Reading Failure, Patrick Groff lists 12 fallacious beliefs that interfere with effective teaching of reading: “It seems incredible that the education establishment could have persisted in the folly of inappropriate reading methodology over so many years and with so many millions of failures. Had we not known how to teach children to read easily and well, this persistence in ineffective methods would have been understandable. However, we have had highly successful methods, programs, and techniques for many, many years . . . [with] conclusive research evidence of their efficacy” says Prof. Barbara Bateman, 1987.

    (I can find and list the 12 fallacious beliefs in a subsequent post if they are not readily available to our readers.)

    What I am pointing out is that there is some deliberate, entrenched stubbornness at play — innocent or malicious — it’s hard to say. It’s harmful, damaging, crippling. When will there be a multi-million dollar damages court award to shake up the teaching “profession”?

    Even Daniel Willingham who is, in my view, trying so hard to be diplomatic regarding the virulent Reading Wars, says that in any list of 16 reading activities (Mind: He does not say checklist.) 20 or 25% of the time should be devoted to phonics, “ . . . when kids are practicing phonics, that practice should be focused.” (pg 82, Raising Kids Who Read, 2015).

    If there isn’t a Teaching Students To Read in Primary Years Checklist, why doesn’t someone produce it?

  4. Tunya Audain says:

    What Happened To The “Effective Schools Checklist” ?

    Ron Edmonds, Harvard 1978, initiated a promising move for systematic adoption of effective standard practice in schools with Effective Schools articles. This checklist emerged.

    Why is it that in aviation and medicine checklists have now become standard practice, but not in other occupations? What’s become of the idea of a basic Checklist in school systems? Atul Gawande, in his book Checklist Manifesto, says that some professionals think it’s beneath them to have checklists. That is a feeble reason to drop a useful tool that’s proven a savior in other fields.

    in view of the OECD’s most recent publication — UNIVERSAL BASIC SKILLS — the Checklist idea needs revival.


    “We can whenever, and wherever we choose, successfully teach all children whose schooling is of interest to us. We already know more than we need in order to do this. Whether we do it must finally depend on how we feel about the fact that we haven’t so far.” (Ron Edmonds, 1978)

    ___ 1. Instructional Leadership — Principal is an effective communicator (with staff, parents, students, school boards), an effective supervisor, & the instructional leader in the school

    ___ 2. Focused School Mission — General consensus by the school community (staff, parents, students) on goals, priorities, assessment, accountability. The mission statement is published and reviewed regularly.

    ___ 3. Orderly Environment — Purposeful atmosphere conducive to teaching and learning.

    ___ 4. High Expectations — Demonstrated high expectations not only for all students but also for staff as well. The belief is that students are capable and able to achieve, that teachers are capable and not powerless to make a difference.

    ___ 5. Mastery of Basic Skills — In particular, basic reading, writing and math skills are emphasized with back-up alternatives available for students with special learning needs.

    ___ 6. Frequent Monitoring of Results — Means exist to monitor student progress in relationship to instructional objectives (and results can be easily conveyed to parents).

    ___ Means to monitor teacher effectiveness

    ___ A system of monitoring school goals

    ___ 7. Meaningful Parent Involvement — Parents are kept well-informed re: programs, goals, etc. There is ample opportunity for them to keep in touch with their child’s progress. They are consulted for feedback about the school and when changes are foreseen. Parent-initiated contact with the school is encouraged.

    ___ 8. Avoidance of Pitfalls — Up-to-date awareness of good educational practice plus retaining currency in the field concerning promising and discredited practices.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s