Bucking the trend – Part I

There were a number of responses on Twitter when Pasi Sahlberg posted a Tweet apparently quoting Diane Ravitch:

At a basic level, the statement contradicts itself. If a student taking the same test at different times will get different results then that student’s results cannot be ordained by their family income and parents’ education. Nevertheless, being charitable, there is a sense in which Pasi and Ravitch are right and we will return to that later.

Looking at the broader picture, there is irony in Sahlberg coming out against standardised testing, if that is what he is doing. Sahlberg is currently professor of education policy at the Gonski Institute for Education at the University of New South Wales in Sydney, but he is originally from Finland. He rose to prominence as an authority on Finnish education, writing books on what the world can learn from Finland. And the world is keen to learn these lessons. Why? In the early rounds of the Programme for International Student Assessment (PISA), Finland gained some of the highest results and this sparked a rush to find out how they did it. But PISA, of course, is a set of standardised tests.

Since those early days, Finland’s PISA performance has significantly declined and the factors often cited as the cause of its early success are likely to be wide of the mark. It still performs relatively well compared to other countries, but this has never been a valid comparison. Countries differ on a variety of factors from the homogeneity of the population, all the way down to the home language and how regular and easy it is to learn. This means that the direction of travel of a particular state or country tells us more than any comparison between different countries.

So in what sense are Sahlberg and Ravitch right about standardised testing? Well, it certainly correlates strongly to family background. In Australia, schools are given an ICSEA score that measures educational advantage. School students also sit standardised NAPLAN assessments in English and mathematics. The correlation between the two is striking (thanks to Julian Rossi, @julianvrossi):

If Sahlberg and Ravitch made the claim that standardised tests are unreliable at the student level but the aggregate scores correlate with family background at the school level, then the claim is more justified. No assessment is ever completely reliable at the student level and therefore accepting their point about reliability depends on how much variation you are prepared to tolerate.

Why would standardised test scores correlate to family background? If all else is equal, it makes sense that children from financially stable homes whose parents are highly educated would do better than those who lack this background, even if there is a fierce argument about how much of this advantage is nature versus nurture. However, in Rossi’s graph, we can clearly see that all thing are not equal. There’s a school with an ICSEA score just over 800 that is far outperforming many schools with an ICSEA score above the average of 1000. I wonder which school this is and I wonder what they are doing?

That’s the advantage of having standardised test scores to consult. We can now ask these questions. Even better, NAPLAN assessments take place at Years 3, 5, 7 and 9 and so, if we wish, we can examine which schools have students who make the most progress.

For instance, Blaise Joseph of the Centre for Independent Studies used NAPLAN data to identify primary schools that are bucking the trend, visited them and described some of the common themes such as strong behaviour policies, explicit teaching and evidence-informed reading instruction. Ideally, it would be good to compare this with a control group of less effective schools*, but nonetheless, this kind of analysis is useful to schools and policymakers who want to know how to improve.

In Part 2, I will examine another context where some schools are bucking the trend.


*You can see why this might be difficult to do. It’s relatively straightforward to call a high-performing school and ask if you can visit them and find out about why they are successful. It is harder to call a low-performing school and ask if you can visit them and find out why they are unsuccessful. Nevertheless, it is feasible that some schools would want to cooperate in order to find ways to improve and this kind of research should be a focus of university education faculties.

4 thoughts on “Bucking the trend – Part I

  1. That dot plot is very, very misleading. It gives an R-squared of 0.8, but that is for the average across the school, and when you average like that you are losing far too much information. It doesn’t show the wealth of an area predicts results — that’s the classic error of suggesting correlation is causation.

    Clever people tend to have clever children and earn more money. Hard working people have hard working children that do better than average and live in nicer areas. That’s all the dot plot shows, since it averages out all other variation. All the rich kids who do badly in a rich area school get averaged out. All the poor kids that do well in a poor school get averaged out. What you are left with the stunningly obvious situation that clever or hard working people have clever or hard working kids who do tend to do well at school.

    That says nothing at all about how the performance of an individual is related to the income of their parents. That dot plot might show a similar trend, but would have a far, far lower R-squared.

    Interesting: what do you think a dot plot of that sort should show in an ideal world?

    The only way it would not be like that is if poor areas drew students who were equally smart and equally hard-working as rich areas and taught them just the same. For that to be true, it would require either a) there is no passing on of brains or work ethic**, or b) that clever hard-working people will not on average earn more and live in richer areas. You actually have to be quite oblivious to the real world to think those things could ever be true.

    (There is another option. Some schools with clever kids might manage to teach them uniformly badly, and some schools with non-hard working kids might somehow work some magic and make them do uniformly well. That would lower R-squared. It is also not even remotely a possibility of ever happening.)

    ** note that this is only on average, because the dot plot is averages. So you have deny any amount of science that intelligence and work ethic are correlated between parents and children. It doesn’t even have to be genetic (although science shows that there is a genetic portion) but could be entirely due to home environment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.