Why the Scientific American article on maths education doesn’t add up

PISA recently released a report about the data that they have collected on maths teaching and learning strategies. I analysed some of this data and related it to the claims that PISA made. The report was quickly followed by an article in Scientific American.

The Scientific American article focused on one area of the PISA report in particular – the rate at which students report using “memorisation” strategies. In the working paper used as a basis for this report, the measure used to quantify memorisation is explained. Students were asked the following questions:

“For each group of three items, please choose the item that best describes your approach to mathematics.

Labels (not shown in the questionnaire): (m) memorisation (e) elaboration (c) control

a) Please tick only one of the following three boxes.

1 When I study for a mathematics test, I try to work out what the most important parts to learn are. (c)

2 When I study for a mathematics test, I try to understand new concepts by relating them to things I already know. (e)

3 When I study for a mathematics test, I learn as much as I can off by heart. (m)

b) Please tick only one of the following three boxes.

1 When I study mathematics, I try to figure out which concepts I still have not understood properly. (c)

2 When I study mathematics, I think of new ways to get the answer. (e)

3 When I study mathematics, I make myself check to see if I remember the work I have already done. (m)

c) Please tick only one of the following three boxes.

1 When I study mathematics, I try to relate the work to things I have learnt in other subjects. (e)

2 When I study mathematics, I start by working out exactly what I need to learn. (c)

3 When I study mathematics, I go over some problems so often that I feel as if I could solve them in my sleep. (m)

d) Please tick only one of the following three boxes.

1 In order to remember the method for solving a mathematics problem, I go through examples again and again. (m)

2 I think about how the mathematics I have learnt can be used in everyday life. (e)

3 When I cannot understand something in mathematics, I always search for more information to clarify the problem. (c)”

I am not convinced that these memorisation options represent actual memorisation strategies. Also, the questions are asked in a way that forces a discrete choice. The accepted practice in psychology is to use a scale of agreement with any given statement (e.g. a Likert scale). Without this, we have a validity and reliability problem. For instance, a student might partly agree with all three responses to question a) but when they are forced to select one response then this will be recorded as 100% agreement with that option and 0% agreement with the alternatives. This is the same reason why the Myers-Briggs personality test is invalid and unreliable.

It is therefore hardly surprising that I could find no correlation between the “index of memorisation” that PISA derive from these responses and a country’s PISA mean maths score. These questions probably do not reliably measure the use of memorisation.

Yet the Scientific American article makes a number of claims about memorisation on the basis of this data. Unfortunately, the authors provide no references and they seem to be in possession of data that is not presented in the PISA report (if either author reads this post then I would be grateful for this data). Nevertheless, I think some of these claims are highly unlikely and I wonder whether the authors may have made an error.

I will list these claims below and then comment on them.

1. In every country, the memorizers turned out to be the lowest achievers, and countries with high numbers of them—the U.S. was in the top third—also had the highest proportion of teens doing poorly on the PISA math assessment.

I cannot tell how a “memoriser” is defined from the PISA report. For instance, is it a person who answers with a class (m) response to all of the questions above, three of them, two of them? Similarly, data on the number of such memorisers in each country is not provided.

I would not be surprised to find out that, in any given country, these memorisers are the lowest achievers but I am not sure what this would tell us. As Robert Craigen points out in a comment on a previous post, memorisers might have resorted to some of these strategies due to poor teaching. They may also have less understanding of, or interest in, the survey questions.

However, I find it highly unlikely that countries with high numbers of memorisers correlate with teens doing poorly on the PISA math assessment. Presumably, countries with higher numbers of memorisers will have a higher overall index of memorisation. If not, this would require the remaining non-memorisers to use far fewer memorisation strategies than the overall mean. If you plot percentage of maths low achievers against index of memorisation then there is no correlation.

2. Further analysis showed that memorizers were approximately half a year behind students who used relational and self-monitoring strategies.

3. In no country were memorizers in the highest-achieving group, and in some high-achieving economies, the differences between memorizers and other students were substantial.

Again, I would like to see the data here but I can believe it.

4. In France and Japan, for example, pupils who combined self-monitoring and relational strategies outscored students using memorization by more than a year’s worth of schooling.

Why select just two countries like this? Again, I don’t have the underlying data but, if I did, it wouldn’t tell us much. It is fraught enough to try to make comparisons across many education systems of different sizes and with different cultures. At least if we include all of them then we might pick up some general trends. I’m sure it would be possible to prove almost anything with just two examples.

5. The U.S. actually had more memorizers than South Korea, long thought to be the paradigm of rote learning.

Again, we would need to know the definition of a “memoriser”.

6. Unfortunately, most elementary classrooms ask students to memorize times tables and other number facts, often under time pressure, which research shows can seed math anxiety. It can actually hinder the development of number sense.

I would love to see this research. Victoria Simms recently reviewed a book by one of the authors of the Scientific American article and found a similar claim:

“Boaler suggests that reducing timed assessment in education would increase children’s growth mindsets and in turn improve mathematical learning; she thus emphasises that education should not be focused on the fast processing of information but on conceptual understanding. In addition, she discusses a purported causal connection between drill practice and long-term mathematical anxiety, a claim for which she provides no evidence, beyond a reference to “Boaler (2014c)” (p. 38). After due investigation it appears that this reference is an online article which repeats the same claim, this time referencing “Boaler (2014)”, an article which does not appear in the reference list, or on Boaler’s website. Referencing works that are not easily accessible, or perhaps unpublished, makes investigating claims and assessing the quality of evidence very difficult.”

7. In 2005 psychologist Margarete Delazer of Medical University of Innsbruck in Austria and her colleagues took functional MRI scans of students learning math facts in two ways: some were encouraged to memorize and others to work those facts out, considering various strategies. The scans revealed that these two approaches involved completely different brain pathways. The study also found that the subjects who did not memorize learned their math facts more securely and were more adept at applying them. Memorizing some mathematics is useful, but the researchers’ conclusions were clear: an automatic command of times tables or other facts should be reached through “understanding of the underlying numerical relations.”

This claim does at least provide a clue as to where to find the evidence although it is a little odd. The neuroscience part of the claim is essentially irrelevant to teachers – why care what ‘brain pathways’ are used? Teachers generally have no opinion on this. We need to focus instead on the quality of learning.

I think I have found the paper. Unusually, it does complete both a neuroscience imaging study and a behavioural study on the quality of learning, as suggested in the Scientific American claim. The participants were 16 university students or graduates. They did a series of trials where they were given two numbers, A and B. In the ‘strategy’ condition, students were given a formula to apply such as ((B-A)+1)+B)=C in order to work out the answer, C. In drill instruction, they were given A, B and the response, C to simply memorise. Surprisingly, the memorisers did pretty well on a later test but, wholly unsurprisingly, they could not extend this to transfer tasks involving new values for A and B. This is entirely consistent with the findings of cognitive load theory were problem solving so occupies our attention that we cannot infer the underlying rule. The strategy example is much more like following a worked example.

However, none of this bears much relationship to memorisation strategies in the PISA report. Is anyone attempting to teach students all of the possible questions that they might be asked and all of the possible numerical answers to these questions? In fact, the use of formulas like in the above “strategy” condition is often criticised as the “rote” learning of formulas and I imagine that this is what maths memorisers – if well-defined – would be trying to memorise.

This research does not seem to apply to the learning of basic maths facts such as multiplication tables. Teachers attempt to teach these to the point of memorisation but the underlying rule is not withheld. Tables are built up from counting patterns, arguments about groups of the same size and so on. Patterns are highlighted like the ones in the 11 and 9 times tables and a few more facts are committed to memory through practice such as 7 x 8 = 56. But these are very simple operations and nothing like the contrivance ((B-A)+1)+B)=C. In fact, the benefit of knowing simple multiplication results ‘by heart’ is that you can then attend to the other elements of a complex operation.

8. Timed tests impair working memory in students of all backgrounds and achievement levels, and they contribute to math anxiety, especially among girls.

This is partially a repeat of claim 6 but also adds the claim that timed tests impair working memory. Again, it would be good to see the evidence to support this.