Five questions to ask an education guru

Imagine that you are a teacher in a school and you have just sat through a presentation by an education guru of some sort. Let’s call her ‘Marion’. If it is safe to do so, what questions should you ask Marion at the end of the presentation? I have a few suggestions.

1. What are you actually suggesting that we should do?

A typical education guru will say plenty of things that don’t actually lead to concrete proposals. She may spin anecdotes about a fantastic school that she recently visited where everything was marvelous.

Alternatively, she may construct terms and then set about defining these terms. You may find yourself hearing about ‘Productive Change Capacity’, how this is made up of constructive professional dialogue, openness to the views of others etc. and how this can be contrasted with ‘Resistive Change Capacity’ which is made of restricted professional dialogue, closed mindedness and so on. I’ve just made-up these two terms but education literature is full of such tautologies that lead us nowhere in particular.

Instead, if Marion really has anything to offer then she should be able to describe what might change as a result of taking on her ideas. Of course, you don’t expect Marion to know everything about your context and so she might need to ask a few questions too. However, she clearly should have something to offer or there really is no point.

2. What problems do your proposals solve?

Presumably, implementing Marion’s ideas will make things better in some way. Otherwise, why would we bother? So it seems reasonable to ask what current or typical problems these ideas will solve. This also places the ideas on a testable basis. If they solve a particular problem then what will that look like? How will we know that the problem is solved? Perhaps our students will read more at home or they might become better mathematical problem solvers. Perhaps they will feel better about school. Some of these things are easier to measure than others but any meaningful change should have observable consequences.

3. What would convince you that you are wrong?

Testability leads to a key principle of science that is largely absent from education discussions; the idea of falsifiability. In fact, it is so absent that you are likely to need to persist in order to get an answer to this question. I would predict that Marion’s first response would be to explain why her ideas are not wrong, why they are well grounded in theory or research or whatever. So you’ll probably need to clarify with a follow-up question.

The problem is that many supposed educational theories can explain all possible sets of circumstances. We will see an example shortly. If Marion really cannot think of anything that would convince her that her ideas are wrong then we have something more akin to an unshakable belief than something based on evidence.

Strong theories are always falsifiable. Their strength comes from the fact that, despite this, nobody has managed to demonstrate that they are wrong. There is a common story that tells of how the evolutionary theorist, J. B. S. Haldane, was asked what would falsify the theory of evolution and he answered that finding fossilised rabbits in Precambrian rocks would do it.

4. Does adopting part of the approach give part of the benefit?

One way that a proposed intervention can become unfalsifiable is when it only works if implemented fully and with 100% fidelity. If it works, great. If it doesn’t work then that is because you didn’t do it properly. Either way, the rightness of the original intervention remains unchallenged.

Something like this happened with a huge differentiated instruction study in the U.S. It didn’t work but the authors concluded that this was because the intervention was not implemented correctly, leaving the principle of differentiated instruction unchallenged.

Now it may be the case that some approaches will work with a perfect implementation and will not work or will cause harm if implemented with anything less than this. If this is true then ask yourself how much practical value there would be in adopting this course of action. It seems unlikely in the extreme that you will get any team of teachers anywhere to implement something with complete faithfulness to the originators’ intentions.

However, if implementing part of the program delivers part of the benefits then this seems a much better prospect. You can imagine different teachers having strengths in different elements, at least to begin with. And as the benefits accrue then you might start to win over the skeptics.

5. What are the negative effects?

If a consultant cannot describe the negative effects of their proposed initiative then this is either because there aren’t any, the consultant is badly informed, the consultant is dishonest or it has never been attempted before and you are the guinea-pigs.

I cannot think of any intervention, even those that I would recommend, that have no negative consequences. For instance, a push for explicit instruction would meet with some teacher resistance that would have to be effectively managed.

No questions asked

Of course, you will not need to ask any of these questions if they are answered in the presentation. Some of the best educationalists that I have seen will preempt most of these points. Dylan Wiliam, for instance, has spent a lot of time thinking about the negative impacts of attempts to embed more formative assessment and has tried to develop programs that provide benefits if implemented only in part. I have heard him talk of teachers aiming to embed one new practice per year.

And we shouldn’t be too harsh. Just because a consultant tells an anecdote, it does not mean that she is wrong. Anecdotes enliven presentations and often make them more bearable. A consultant promoting a genuinely effective approach might not have been challenged by such questions before; the education community is often too polite and credulous.

However, if after a thoughtful pause, you don’t get a straight answer then I’d be keen to investigate further before plunging into the next whole-school initiative.

“We’ll need to have a think about that, Marion.”

If you are interested in reading more about the evaluation of educational initiatives and ideas in a way that is accessible to teachers then I recommend Dan Willingham’s book, “When can you trust the experts?”


17 thoughts on “Five questions to ask an education guru

  1. These are also excellent questions to ask ourselves when considering adopting a new program or methodology. Often, the “sell” comes from many directions and we have to synthesise it first.

  2. Stephen Gorard says:

    My question woudl be something like:
    If the approach you (the guru) propose does not actually work how else and how easily could we explain the evidence you presented for the approach? This goes further than consideration of being wrong, to include warrant, parsimony and plausibiity of the claims.

  3. David says:

    The problem we encounter at my school is that while the educators are usually very skeptical, the administration is not. The guru has already gotten a good chunk of change just for being in front of the teachers in the first place–implementation of the “system” means additional money, but is often just gravy for having gotten into the school. We had a set of gurus come and speak to us about their “educational ninja” system–we were all required (though, thankfully, no one bothered to check) to log-in to their webpage after their absurdist presentation (where our principal was smiling and nodding sagely throughout). There were number of tracks and one gained in ninja levels the more proficitent one became (i.e., the school paid to have us be able to access this system; one became more “proficient” by watching a webinar and answering a silly quiz). The program was discontinued due to lack of use (i.e., my school was full of lowly assistant pig-keeper ninja recruits).

    What is actually more evil than the guru is Google–they get teachers who have drank the Kool Aid to visit schools on their behalf to laud the glories of Google Apps for Ed. The concept is similar to what Lululemon Athletica (a US purveyor of yoga clothing) where they “influence the influencer”–by getting instructors on-board as enthusiasts and letting them sell the product. In this way, it becomes a different dynamic than the guru presentation–the presenter is not some business-y person with aproduct to sell, but “one of you”.

    The Google teacher-acolyte who spoke to us a few years ago wasn’t even paid other than travel expenses and the free lunch we gave him. I think Google threw him some change occassionally for representing them.

    Hint to fellow teachers: NEVER become an enthusiast for something derived by a for-profit corporation.

    • Ooooooooooooo David. WELL PUT!! You have expressed everything most of us lowly grunt teachers in the trenches feel. Your description of your principal made me laugh because I have seen that SAME behavior i principals and administrators in my district. Ha ha ha ha ha

  4. In my view, if an intervention shows a clear negative effect unless implemented with 100% fidelity, then schools should RUN HARD IN THE OTHER DIRECTION AND DON’T LOOK BACK.

    And this is granting that the pot of gold may actually exist at the end of the 100% fidelity field of shamrock. Because what caring school would wager the welfare of their students against the unlikely possibility of hitting such a fragile sweet spot of implementation, with real teachers, real infrastructure, real social environment and real students?

    Run. Hard.

  5. Pingback: OTR Links 09/27/2015 | doug --- off the record

  6. Pingback: Does Lao pass the Ashman Test? | Tao Te(a)Ching

  7. Pingback: There are many ways to mess things up | Filling the pail

  8. Pingback: My five favourite blogs of 2015 | David Didau: The Learning Spy

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.