Tackling bias

Embed from Getty Images

We all suffer from a variety of unconscious biases. Further depressing evidence of this has come with the release of a new review by Paul Connolly and colleagues into setting practices from the UK. In England, setting means the process of grouping students based upon prior attainment. For instance, students with high maths attainment would go into one maths class and students with lower attainment would go into a different class. The idea is that this enables the teacher to better target the teaching to the needs of the students. I am ambivalent about setting, even though, as a head of science and a head of maths, I have made use of the practice. Why? Firstly, I have noticed a tendency for the higher attaining groups to end-up being taught by the senior and more experienced members of the department and the lower attaining groups to be assigned new and less senior teachers. Secondly, the process of assigning students to these groups can often be somewhat opaque, with nebulous teacher judgement being a factor in these assignments. This can lead to perverse consequences, as I have written about before. I suspect that it is the range of such real-life effects that leads to mixed results in the research on setting.

Connolly et al. compared students scores in a standardised assessment of maths at the end of primary school with the set they were placed in during the first year of secondary school. As we might expect, these do not exactly match. There is no reason to assume that they should match because performance is likely to change over time and be dependent on different content. Nevertheless, the reviewers describe the difference between the set implied by the standardised test score and the set students were actually placed in as a ‘misallocation’ and the depressing part is that there appear to be some stark systematic differences in misallocation between different groups of students. Girls were 1.5 times as likely as boys to be misallocated to a lower set than their standardised test scores suggested, Black students were 2.4 times more likely that white students to be misallocated to a lower set and Asian students (which tends to mean South Asian students in the UK) were 1.7 times more likely to be misallocated to a lower set. Not good.

However, I would suggest part of the solution also lies within this research. How did the researchers infer misallocation? From standardised test scores. Why does this make sense? Standardised test scores are far less prone to human biases. They are not perfect, by any means, but they are the least biased of all the possible alternatives. Portfolio assessment? You can get a tutor to help with that if you have the cash. Presentation? A whole host of biases come in to play. However, with a standardised assessment, particularly if anonymised as most assessments are during the scoring process, there can only be bias based upon handwriting or certain turns of phrase. In the case of a multiple choice assessment, there is no potential for bias at all.

Clearly, standardised test scores will still reflect differences in preparation for the assessment, and these may reflect wider societal biases and privileges. However, in a world where all assessments of learning are biased to some extent, standardised assessments are clearly the least biased. This is why high stakes appointments and admissions criteria should be more not less dependent on standardised test scores. Once you start taking into account portfolios, presentations, course work, work experience and all of those other factors we are often asked to consider, you create a system that can be more easily gamed by the rich and privileged.

This is why the ancient Chinese invented standardised tests in the first place.

Yes, there are reasonable and valid concerns about the way that certain forms of standardised tests may distort teaching. There should be a robust debate around these issues – a debate I am happy to participate in. What is clear is that anyone who is concerned about bias in education cannot reconcile such a position with an opposition to standardised testing.

2 thoughts on “Tackling bias

  1. Assessment was a major issue in the deliberations of Ofsted’s Maths Advisory Group, and I strongly supported the view that schools should determine their own progression and use objective written assessments to ensure that pupils had achieved a sufficient degree of automaticity before moving on to more complex material. In general, I think it is a very good idea for teachers design their own tests, so long as their objectives are coherent.

    This wasn’t good enough for some members of the group who opposed any kind of objective quizzes or tests. They quite honestly believed that teaching should impart some ineffable abilities that were impossible to detect objectively, but which a properly-trained (ie, NCETM) teacher could sense floating around in the ether.

    Standardised tests may well be the only answer for fair high-stakes testing, but even there we have to beware of the tendency to design them more as psychometric tests than tests of learning.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.