PISA data on maths memorisation

This is Part 1 of a sequence of posts. You can find Part 2 here and Part 3 here.

I recently had the following graph Tweeted into my timeline:

It is from the new PISA report, “Ten questions for mathematics teachers… and how PISA can help answer them.” It is an interesting report containing links to the data set.

The first thing that strikes me about the graph is that it says very little. There is not much correlation between the two measures and neither of them is a measure of maths performance. So what are we meant to conclude?

PISA asked students a number of questions and then developed an “index of memorisation”. For some reason, that’s not quite what has been plotted on the y-axis of their own graph (more later) but I decided to plot 2012 Maths PISA scores against this index. This is what I found:

This is not a strong correlation, implying that either the degree of memorisation does not affect maths outcomes (maybe something like teacher quality swamps it) or that the construct that PISA have used – their “index of memorisation” – is flawed. Have a look at how they calculated it:

“To calculate how often students use memorisation strategies, they were asked which statement best describes their approach to mathematics using four questions with three mutually exclusive responses to each: one corresponding to a memorisation strategy, one to an elaboration strategy (such as using analogies and examples, or looking for alternative ways of finding solutions) and one to a control strategy (such as creating a study plan or monitoring progress towards understanding). The index of memorisation, with values ranging from 0 to 4, reflects the number of times a student chose the following memorisation-related statements about how they learn mathematics:

a) When I study for a mathematics test, I learn as much as I can by heart.

b) When I study mathematics, I make myself check to see if I remember the work I have already done.

c) When I study mathematics, I go over some problems so often that I feel as if I could solve them in my sleep.

d) In order to remember the method for solving a mathematics problem, I go through examples again and again.

Statement a) assesses how much students use rote learning, or learning without paying attention to meaning. The remaining three statements come close to the ideas of drill, practice and repetitive learning.”

The values that you get from this index range between about 0.9 and 1.6 out of 4. So that doesn’t strike me as a huge variation. I am not at all sure that questions a) to d) represent my personal concept of memorisation. And remember, students had to pick one of three options to each question. So they might not have been that enthusiastic about the one they chose. We don’t know exactly what the alternatives looked like but it might have been a case of rejecting those alternatives rather than positively selecting statements a) to d). [It would probably have been better to use questions with the stem, “To what extent..” and a Likert scale – the PISA approach reminds me of one of the flaws in the Myers-Briggs personality test].

We also have to bear in mind that these questions would have been translated into many different languages. The phrase “I learn as much as I can by heart” is quite idiomatic. It has positive connotations in English but do those translate? Was a different idiom used?

Instead of plotting their own “index of memorisation”, PISA seem to have plotted the percentage average of how often a) to d) were selected by students*. This has the effect of changing the rankings quite a lot. Four of the top five on their plot are countries where students will have received these questions in English, which further supports the hypothesis that language might have something to do with it.

PISA further slice and dice their analysis by examining question a) only i.e. the question about learning ‘by heart’. So I plotted this against PISA 2012 Mean Maths Score:

The R-squared value is about 0.00009. Which is low.

In their report, the PISA authors attempt to construct the narrative that countries in the Far East are not as teacher-directed or reliant on memorisation as we might have assumed. There’s a little vignette about Japan and its “zest for living” reforms. This is interesting because I predict that when the PISA 2015 results are released in December, we will see a further entrenchment of Far East nations at the top of the table. Until now, educationalists in places like Australia have tended to be selectively blind to this, preferring to point to the examples of Finland and Canada – the Far East has been dismissed due to cultural differences.

Yet, Canada and Finland have both embarked upon reforms since 2000 that, in my view, have already impacted negatively on their international performance. If this trend continues in PISA 2015, as seems likely, then it may be harder to ignore the performance of the Far East. So perhaps we’re being subjected to a little early spin.

Note: Clicking on the graphs makes them larger and easier to see. You can find the source data here.

Update: after publishing this blog post I noticed that this data set is being used in an article in Scientific American, alongside some interesting neuroscience claims.

*this is not the case – see the later posts

This is Part 1 of a sequence of posts. You can find Part 2 here and Part 3 here.