Evidence and Integrity

The room was packed with math faculty from across the district. The superintendent rose, “I’d like to introduce Jessie. She was recently recognized as the NAMT Maths Innovator of the Year…”

The crowd clapped and some of them whooped.

“Jessie is going to talk to you about how she flipped her math class.”

Jessie approached the podium. The clapping died-down. “I was tired of regular math class,” She explained, “I thought there has got to be a better way than rote memorization of disconnected facts and procedures. So I thought, hey, why don’t I try flipping the classroom? So I suspended all the students upside down from the ceiling.”

An image appeared on the screen of smiling, inverted students.

“I have never looked back. My students love it.”

A senior math teacher with a gray beard and a bald head was sat in the second row. He had been largely unmoved by the presentation so far. Irritably, he raised his hand.

The superintendent called on him, somewhat impatiently, “Yes, Mike. What is it?”

Mike spoke, “Where is the evidence? Where is the evidence that this works?”

Jessie smiled, “There’s plenty of evidence. For instance, we know from brain imaging studies that blood flow increases in the brain during mathematical problem solving and suspending students upside down increases blood flow to the brain. We also know that students have to demonstrate grit and resilience in order to solve problems whilst inverted and these traits have been associated with enhanced life outcomes. We also find that the fact that the writing on the whiteboard is inverted from the students’ perspective means that they have to work harder to process this information and this introduces desirable difficulties that have been shown to improve learning over time.”

Evidence for what?

The example above is deliberately absurd but I have sketched it in order to make an important point. Jessie is not providing evidence for her approach, the central component of which is to suspend students upside down. She is presenting evidence that might be relevant but it is only incidental (and the first point also represents a correlation-causation fallacy). The sort of evidence that would support her method would be a trial where students are randomised into upright versus inverted conditions and their performances compared.

This is important because we often see such reasoning in education. Sometimes, ideas are presented with no reference to any supporting evidence at all, but we also see the situation arise where evidence is presented that is incidental to the main claim that is being made.

I have repeatedly contended that Dan Meyer has ‘no evidence’ for the claim he makes in his famous TED talk. In the talk itself, he chooses not to refer to any evidence and the impression is given that this is just his personal view. The major thrust of his argument is that scaffolding must be removed from the problems that students are presented with in maths:

“So 90 percent of what I do with my five hours of prep time per week is to take fairly compelling elements of problems… from my textbook and rebuild them in a way that supports math reasoning and patient problem solving. And here’s how it works. I like this question. It’s about a water tank. The question is: How long will it take you to fill it up? First things first, we eliminate all the substeps. Students have to develop those, they have to formulate those. And then notice that all the information written on there is stuff you’ll need. None of it’s a distractor, so we lose that. Students need to decide, “All right, well, does the height matter? Does the side of it matter? Does the color of the valve matter? What matters here?” Such an underrepresented question in math curriculum. So now we have a water tank. How long will it take you to fill it up? And that’s it.”

I have taken issue with these suggestions based upon my understanding of cognitive load theory. I think they are misguided. However, this is a genuine criticism of a set of claims and not a personal attack on the individual who is making them.

Since I first wrote about this topic, Dan has disputed my contention that there is no evidence for these ideas. He suggested three sources of evidence which I then took the time to analyse in this post. Briefly, they are:

  • Mayer’s work on multimedia learning which suggests the use of arresting graphics, videos and so on. Strangely, this work also contradicts Dan’s assertions in the TED talk. For instance, Mayer states, “If the message lacks guidance for how to structure the presented material, the learner’s model-building efforts may be overwhelmed.”
  • David and Roger Johnson’s case for the use of ‘constructive controversy’ in social studies lessons.
  • Kasmer and Kim’s work on giving students the opportunity to make predictions.

All of this is incidental to the main claim. Even if you accept the principles outlined in these papers then there is no reason why they could not all be encompassed in an entirely different teaching method to the one that Dan suggests. And so I have maintained my position that there is ‘no evidence’ for this method.

A question of integrity

This has not played well with with Dan Meyer. He has publicly questioned my integrity for maintaining this position.

Meyer Integrity

I think we are in danger here of personalising what should be an academic discussion.

When I claim that Dan has no evidence for this position, it is not a personal attack. It is my sincerely held view based on the reasons I have given above. Dan is entitled to hold a different view. He is entitled to his beliefs and he is entitled to promote those beliefs. In fact, I would argue that Dan’s ideas about maths teaching are pretty mainstream among maths educationalists, even if they are not common practice in classrooms. But you can’t make something as high profile as a TED talk (viewed over 2 million times) that has significantly influenced the discussion of maths teaching and allow for no criticism.

There is also a strong current in education that actually rejects the use of the sort of evidence that I am looking for. Proponents of this view would argue that social sciences such as education are qualitatively different to physical sciences and that seeking empirical evidence is ‘positivism’. So there are legitimate alternatives to my take on this issue that don’t require us to impugn anyone’s character.

It’s fine to disagree. In fact, it is often how we learn. Sometimes it is painful. I am frequently wrong and I don’t like the cognitive dissonance this brings any more than anyone else does. But let’s try to keep it civil.


8 Comments on “Evidence and Integrity”

  1. Dan’s view was one championed also by the late Grant Wiggins. And the idea lives on. I’m going through a math textbook that the school I work in uses: the Big Ideas math series. (U.S). It precedes each lesson with an activity in which students are expected to discover key ideas that are then presented the next day in direct style. Some of the activities are extremely vague, so it is not exactly clear what is to be discovered. (E.g., “find a relationship between the sides of the right triangle based on this activity”. They’re supposed to discover Pythagoras’ Theorem: a^2 + b^2 = c^2. And then in the problem sets, there are always a few problems for which the methods and concepts are not yet discussed–they are several chapters ahead. Lack of scaffolding supposedly builds “deep understanding”, whereas scaffolding ends up with “bored” students who learn the material “by rote”.

    • gregashman says:

      I had a number of debates with Grant Wiggins. He was always a generous opponent. We chiefly disagreed over Hirsch’s ideas on knowledge and reading comprehension but he did also display an instrumental view of mathematics – I intend to blog about this kind of view soon.

      The textbook model that you are describing sounds a lot like Manu Kapur’s ‘productive failure’. There is some evidence for this and Dan Meyer has quoted it from time-to-time (although it is a different model to the one he proposes in his TED talk). I am underwhelmed by the strength of this evidence, however, and the list of conditions that Kapur states are necessary to make it work.

      • Productive failure is something I’ve used (at least I think I have) but in small increments. It follows along the lines of ZPD and scaffolding. If we covered factoring trinomials in which all terms are positive, then the next day I introduce a problem in which some of the terms are negative. Or the lead coefficient is > 1. Since they’ve already gotten the hang of factoring the easy trinomials, some students will make the stretch and figure out how to do the more complex ones. Others may not figure it out but their interest is piqued from trying so they are more receptive for the instruction that follows. This situation may be the “evidence” you are saying exists for productive failure.

        But how I see it implemented is that students are expected to make giant leaps, and the problems with which they are confronted are often beyond their ZPD. Supposedly they then are receptive to learn key concepts/skills in a “just in time” manner. This is markedly different than what I described above, and more than likely results in cognitive overload.

        I agree that Grant was very accommodating, I think he nevertheless was insistent on “no scaffolding” and his approaches were short-sighted.

  2. Tempe says:

    Yes, I agree. Whenever I attempt to dispute an assertion someone has made about education it usually ends with some abuse. I find it interesting as it suggest to me that they may not feel they have a strong enough argument so resort to name-calling in order to shut the conversation down. Also. evidence, if ever presented, is often very flimsy. Then the cry is, “why should I have to use the evidence you ask for I know what works in my class room and you’re not a teacher so be quiet.”

  3. Janita says:

    Ad hominem attacks like this one on you, Greg, are not only rife these days but are apparently regarded as acceptable. It occurs to me that the current enthusiasm among humanities teachers for promoting “critical thinking” in fact rewards students who resort to ad hominem arguments. Instead of dealing with the substance of a text, students are encouraged to ask questions about the “subtext”: What is the writer’s sex/age/religion/political position? Whose interests does he/she represent? This is what passes for critical analysis.

    When I was young, I discovered a book that thrilled me — Straight and Crooked Thinking, by Robert Thouless. It listed common logical fallacies, including such dishonest tricks
    as ad hominem attacks, and how to rid them from your thinking. First published in 1930 (republished in 1953), it would no doubt be dismissed as antiquated by the post-modern generation, but like any good book its relevance is perennial.

    • Pique Boo says:

      That book looks useful. With a whizzy (esp. maths) child in a comp Y8 we have some space for a few things she won’t get at school. I’ve long had formal/informal logical fallacies on the list because she’s one of life’s natural truth seekers, not a good post-modern child. I wonder if there’s a correlation between the latter and ‘maths whizzy’?

  4. […] Dan Meyer is well-known in the field of maths education. He has made strong claims – notably in a popular TED talk – that we need to change the way that we teach maths. I have criticised these claims as lacking evidence and Meyer has disputed this.  […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s