Evidence and IntegrityPosted: July 10, 2016
The room was packed with math faculty from across the district. The superintendent rose, “I’d like to introduce Jessie. She was recently recognized as the NAMT Maths Innovator of the Year…”
The crowd clapped and some of them whooped.
“Jessie is going to talk to you about how she flipped her math class.”
Jessie approached the podium. The clapping died-down. “I was tired of regular math class,” She explained, “I thought there has got to be a better way than rote memorization of disconnected facts and procedures. So I thought, hey, why don’t I try flipping the classroom? So I suspended all the students upside down from the ceiling.”
An image appeared on the screen of smiling, inverted students.
“I have never looked back. My students love it.”
A senior math teacher with a gray beard and a bald head was sat in the second row. He had been largely unmoved by the presentation so far. Irritably, he raised his hand.
The superintendent called on him, somewhat impatiently, “Yes, Mike. What is it?”
Mike spoke, “Where is the evidence? Where is the evidence that this works?”
Jessie smiled, “There’s plenty of evidence. For instance, we know from brain imaging studies that blood flow increases in the brain during mathematical problem solving and suspending students upside down increases blood flow to the brain. We also know that students have to demonstrate grit and resilience in order to solve problems whilst inverted and these traits have been associated with enhanced life outcomes. We also find that the fact that the writing on the whiteboard is inverted from the students’ perspective means that they have to work harder to process this information and this introduces desirable difficulties that have been shown to improve learning over time.”
Evidence for what?
The example above is deliberately absurd but I have sketched it in order to make an important point. Jessie is not providing evidence for her approach, the central component of which is to suspend students upside down. She is presenting evidence that might be relevant but it is only incidental (and the first point also represents a correlation-causation fallacy). The sort of evidence that would support her method would be a trial where students are randomised into upright versus inverted conditions and their performances compared.
This is important because we often see such reasoning in education. Sometimes, ideas are presented with no reference to any supporting evidence at all, but we also see the situation arise where evidence is presented that is incidental to the main claim that is being made.
I have repeatedly contended that Dan Meyer has ‘no evidence’ for the claim he makes in his famous TED talk. In the talk itself, he chooses not to refer to any evidence and the impression is given that this is just his personal view. The major thrust of his argument is that scaffolding must be removed from the problems that students are presented with in maths:
“So 90 percent of what I do with my five hours of prep time per week is to take fairly compelling elements of problems… from my textbook and rebuild them in a way that supports math reasoning and patient problem solving. And here’s how it works. I like this question. It’s about a water tank. The question is: How long will it take you to fill it up? First things first, we eliminate all the substeps. Students have to develop those, they have to formulate those. And then notice that all the information written on there is stuff you’ll need. None of it’s a distractor, so we lose that. Students need to decide, “All right, well, does the height matter? Does the side of it matter? Does the color of the valve matter? What matters here?” Such an underrepresented question in math curriculum. So now we have a water tank. How long will it take you to fill it up? And that’s it.”
I have taken issue with these suggestions based upon my understanding of cognitive load theory. I think they are misguided. However, this is a genuine criticism of a set of claims and not a personal attack on the individual who is making them.
Since I first wrote about this topic, Dan has disputed my contention that there is no evidence for these ideas. He suggested three sources of evidence which I then took the time to analyse in this post. Briefly, they are:
- Mayer’s work on multimedia learning which suggests the use of arresting graphics, videos and so on. Strangely, this work also contradicts Dan’s assertions in the TED talk. For instance, Mayer states, “If the message lacks guidance for how to structure the presented material, the learner’s model-building efforts may be overwhelmed.”
- David and Roger Johnson’s case for the use of ‘constructive controversy’ in social studies lessons.
- Kasmer and Kim’s work on giving students the opportunity to make predictions.
All of this is incidental to the main claim. Even if you accept the principles outlined in these papers then there is no reason why they could not all be encompassed in an entirely different teaching method to the one that Dan suggests. And so I have maintained my position that there is ‘no evidence’ for this method.
A question of integrity
This has not played well with with Dan Meyer. He has publicly questioned my integrity for maintaining this position.
I think we are in danger here of personalising what should be an academic discussion.
When I claim that Dan has no evidence for this position, it is not a personal attack. It is my sincerely held view based on the reasons I have given above. Dan is entitled to hold a different view. He is entitled to his beliefs and he is entitled to promote those beliefs. In fact, I would argue that Dan’s ideas about maths teaching are pretty mainstream among maths educationalists, even if they are not common practice in classrooms. But you can’t make something as high profile as a TED talk (viewed over 2 million times) that has significantly influenced the discussion of maths teaching and allow for no criticism.
There is also a strong current in education that actually rejects the use of the sort of evidence that I am looking for. Proponents of this view would argue that social sciences such as education are qualitatively different to physical sciences and that seeking empirical evidence is ‘positivism’. So there are legitimate alternatives to my take on this issue that don’t require us to impugn anyone’s character.
It’s fine to disagree. In fact, it is often how we learn. Sometimes it is painful. I am frequently wrong and I don’t like the cognitive dissonance this brings any more than anyone else does. But let’s try to keep it civil.