Last week saw the release of the Programme for International Students Assessment (PISA) collaborative problem solving results. This was accompanied by headlines suggesting that Australia and the U.K. had performed particularly well in these assessments.
I’m not sure about this.
In PISA 2015 science, Australia was ranked 14th. On the recent problem-solving test it was ranked tenth. This seems like an improvement. However, you have to take into account the number of countries that participated. In the science tests, 72 countries took part whereas only 51 took part in the collaborative problem solving test. The Australian performance of coming tenth out of 51 is roughly equivalent of coming 14th out of 72. So it’s pretty much the same. Similarly, the U.K. came 15th out of 51 in collaborative problem solving which is the equivalent of coming 21st out of 72; a worse performance than in PISA 2015 science where the U.K. came 15th out of 72.
We could perhaps argue about this. For instance, if the countries involved in the collaborative problem solving test were generally stronger PISA performers than the ones left out then this might make a difference. Perhaps. I haven’t run the numbers.
But all of this fades when we consider the fact that collaborative problem solving is not even a thing.
The OECD report notes that the countries who perform best are those that also top the charts for maths, reading and science. It states:
“Collaborative problem-solving performance is positively related to performance in the core PISA subjects (science, reading and mathematics), but the relationship is weaker than that observed among those other domains.”
There is no discrete skill of problem solving that can be taught in schools. Schools can only teach solution approaches to specific kinds of problems. There are clearly differences in individuals’ problem solving performance but these will be related to general intelligence and domain knowledge. For instance, solving a science problem will depend on science knowledge. This is why PISA’s test of collaborative problem solving skills correlates with a country’s performance in science, reading and maths.
In PISA’s bizarre assessment, students had to solve problems by interacting with imaginary, computer generated agents via a computer. So this would also have been affected by various cultural factors related to how comfortable and familiar students were with such an approach. Essentially then, PISA’s test measures a messy conflation of content knowledge and cultural factors.
So it really tells us nothing at all.