PISA evidence for project-based learning in maths

The more I analyse the PISA data set, the more I am surprised by what PISA have chosen to highlight in their recent report. The PISA measure of memorisation – a topic stressed by PISA and by Boaler and Zoido in a follow-up article – hardly correlates with PISA maths results at all. Teacher-directed instruction and student-oriented instruction appear to be more significant.

One of the questions asked by PISA in the category of “student-orientation” is about how often a student’s maths teacher, “Assigns projects that require at least one week to complete.”

This seems to correlate negatively with the PISA 2012 mean maths score.

Should we therefore avoid project-based learning in mathematics?

Obviously, there are a number of limitations to this analysis. Can we really compare jurisdictions like this? And as Ben Wilbrink pointed out on Twitter, students are not simply influenced by what their current teachers are doing but also a great deal by what their previous teachers did. Nevertheless, I am following the kind of analysis that PISA have themselves chosen to highlight.

We could perhaps address the first of these points – comparability across systems – by looking at the students in each jurisdiction and comparing the way that their maths performance relates to these various different measures. This is what Caro, Lenkeit and Kyriakides did in a paper published earlier this year (thanks again to @cbokhove for the link).

They did not separate the analysis into individual questions but looked at the construct of student-orientation as a whole, using PISA’s index. For every education system, they found a negative relationship between maths scores and the student-orientation measure: “…model estimates produce a decisively negative association with student-oriented instruction across education systems.” So this seems to broadly agree with my between-country analysis.

Caro et. al. also looked at a number of other measures in this way. For instance, they examined measures of classroom climate (note that they analysed 62 systems in total, omitting Albania and Lichtenstein for methodological reasons):

“For the instructional context variables we found a positive association with disciplinary climate in 61 education systems and a positive association with classroom management in 47 systems. Teacher student relations were positively related to mathematics performance in 16 systems and negatively in 22 systems.” [condition codes omitted]

This is interesting. I would expect classroom management and discipline to correlate positively with performance but I would have thought that teacher student relations would also correlate positively. It might be worth examining the questions that PISA asked in order to come up with this measure.

They also looked at the teacher-directed and cognitive-activation measures that I investigated across countries. I found a weak, negative correlation between both of these and maths performance (here and here). Caro et. al. decided to try to fit curves to these relationships rather than straight lines, on the plausible assumption that there are diminishing returns to be had by further increasing the use of a strategy once there is already a lot of it going on. Across education systems, they found*:

“…mathematics performance tends to improve for higher levels of cognitive activation but at a decreasing rate or even with negative associations for very high frequencies of cognitively activation activities. In other words, for students who report the most frequent use of cognitively activating activities from their teachers, the initially positive association of this strategy stagnates or becomes even negatively associated with performance…

…Results are similar for teacher-directed instruction… where the association with mathematics performance is positive at fewer frequencies, decreases when teacher-directed instruction is employed more often and ultimately becomes negative at higher frequencies. Again, in the great majority of systems there is a positive side of teacher-directed instruction that is underestimated if guided solely by linear associations.”

Examining these graphs by eye is not massively convincing. They appear to me to show little indication of any relationship at all. I suspect that the student-orientation construct is capturing something real about what is happening in lessons; something that impacts upon performance. I am not sure that these others do.

*have a look at how these findings are reported in the paper’s highlights section

Advertisement