What will we learn from the new round of international test results?

Embed from Getty Images

At the end of this month, TIMSS and PIRLS results are released and at the start of December we will also have new PISA results. All of the assessments were sat in 2015. What will they show?

Firstly, I don’t think it is particularly useful to look at ranks of countries. These can fluctuate due to tiny changes in scores as well as changes in the list of countries participating.

There is more value in comparing general trends across countries on scores rather than ranks, particularly if we examine one subject area such as maths or reading. 

Yet probably the most useful analysis is to look at how an individual country has changed relative to its previous scores and then see if this correlates to changes in demographics or education policy.

TIMSS and PIRLS will have less fanfare than PISA. TIMSS assesses maths and science in grades 4 and 8 and PIRLS is a test of reading in grade 4. East Asian countries tend to dominate TIMSS. The test questions are pretty standard when compared with PISA questions. PISA items often involve a lot more context and therefore conflate measures of science and maths with reading ability. 

Finland skipped TIMSS between 1999 and 2011. In 2011 it didn’t perform particularly well so it will be interesting to see the trend from 2011 to 2015. My guess is that it will be stable or downward.

I mention Finland because its PISA results have made it the darling of worldwide education and have made a star of Pasi Sahlberg, a Finnish educationalist. For many, the attraction of Finland is that it seems to combine test success with progressive education, although this is not all that it seems. Clearly, what Finland chooses to do now cannot be the cause of past successes and some of its more recent choices – such as the new phenomenon based learning – are likely to lead to a decline in outcomes. 

This is why I am predicting a PISA decline for Finland. I am also predicting a similar decline, particularly in maths, for Canadian provinces that have previously done so well. This is due to a pivot towards constructivist maths teaching in recent years. Despite getting a shock in 2012, I’m not sure that Canadian politicians have adequately addressed this or that the effects have been mitigated in schools.

As these countries exit the stage, PISA results will start to look more like TIMSS. East Asian countries will dominate even more than they do at present. 

The early spin suggests that this is likely to be the case. We’ve heard a lot about East Asian countries not prioritising memorisation. This is based upon survey data in which I can find no pattern at all. So why focus on this? When the results are out we might find people arguing that anglophone countries should teach maths and science using more East Asian methods. So it may be an early attempt to control that argument.

The only very clear signal from this survey data was that strategies that PISA has defined as student-oriented – practices such as project-based learning – seemed to have a clearly negative impact on outcomes. Which must cause a little cognitive conflict over at PISA headquarters where Andreas Schleicher is advocating these very same methods.

There is an innovation to PISA this year in the form of a test of ‘collaborative problem solving’. It’s hard to tell what this will show although one thing is certain: it is not a test of collaborative problem solving because no such general skill exists. Instead, it will be a measure of how well students can complete a collaborative problem solving test as designed by PISA. It seems likely that this will correlate with the other PISA measures, adding little extra information.

Advertisements

4 Comments on “What will we learn from the new round of international test results?”

  1. paceni says:

    What, if anything, does PISA measure? This article reads like an apologist’s plea on behalf of a fundamentally flawed instrument.

  2. Tempe says:

    I agree with your predictions. I wonder what it will take for the govt. in Canada to see that there might be a correlation between the declining results and the newer methods they adore. I won’t hold my breath.

  3. paceni says:

    Greg,
    It would be vital to highlight a fairly obvious point – Pisa is not and never will be like TIMSS. Schleicher & Pisa have clung desperately to IRT and the Rasch model in particular. Unfortunately Schleicher has been unable to unwilling to tell the world exactly what his instrument measures. Perhaps you can tell us what Pisa measures. The truth is that Pisa is all about flawed league tables, used to panic and control participants, but producing nothing useful Their relationship with ETS will not save them. OECD Pisa, just like the EU project is dead but not yet buried. Don’t prolong the agony, cease your futile reccussitation efforts and let OECD Pisa die in peace.
    The world has nothing to learn from Pisa except how gullible education ministers are in their quest for simple answers to complex problems. Still waiting to learn how OECD Pisa can sum up a nation’s education system in 3 numbers.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s