How do we know that our students are making progress?

Embed from Getty Images

If I said to you that School X obtains higher academic outcomes than School Y, you may reasonably ask, ‘So what?’ Anyone who has been involved in education for more than five minutes knows that it is not as simple as this. Some schools are selective and some draw their students from affluent communities. These factors could make obtaining higher scores easier. Similarly, if a school draws its students from a disadvantaged population then obtaining high scores may be far harder. This is not a fair comparison.

It seems far better to focus on progress. If we can somehow map the progress of students as they pass through a school, we will have a measure that we can assume is more closely related to the quality of the teaching. So progress it is.

But what is progress? This is not a simple idea and I think it can lead us in to some unhealthy practices. A seemingly straightforward way to measure progress may be to ask students to repeat the same task at regular intervals and record whether their skill level improves. This is essentially what happens with the NAPLAN writing tasks in Australia. The danger, however, is that it narrows the curriculum. Schools may feel an incentive to endlessly practice narratives and persuasive pieces set in banal contexts. The use of banal contexts, such as eating your greens or a pet going missing, ensures that all students have sufficient knowledge to attempt the task.

This is clearly an issue for those of us who prioritise a knowledge-rich curriculum. In microcosm, we know that possession of broad background knowledge aids reading comprehension and this is vital because reading is a key tool of academic learning. We can perhaps best understand this by invoking constructivist learning theory. Constructivism posits that, “…learners interpret new information using knowledge that they have already acquired. Learners activate prior knowledge and try to relate new information to knowledge they already possess.” This is often understood to mean that we must place learning in contexts that are somehow ‘relevant’ and ‘authentic’, but a alternative interpretation is that we must concern ourselves with the quality of prior knowledge that a student brings to any new situation by teaching it to them in advance. This will then affect the way we structure the curriculum.

However, the perceived need to repeatedly cycle through different iterations of the same task set in a banal context is in opposition to building broad knowledge. Even if you try to fuse the two by, for example, asking students to write persuasively in each new content domain they encounter, you will find that the quality of their performance will depend upon their understanding of the domain and that, in turn, will depend upon the complexity of ideas in that domain. Your attempts to measure progress in ‘writing persuasively’ will founder as scores seemingly rise and fall at random. Over time, the implicit incentive to pick less complex domains may lead to the degrading of your original curriculum objectives.

But is it true to infer that no progress has been made as these skill scores rise and fall? Let me ask it this way: If a student learns about the Romans in Term 1 and then learns about the Vikings in Term 2, have they made progress (assuming they do actually learn this content)? It may be the case that their persuasive writing skills have not improved, but can they not progress in knowledge as well as skills? If we think of the eventual destination as being a broad knowledge of the world that will facilitate future learning and active participation in civil society, then are we not a step closer? Is that not progress? In think it is, but I don’t know how to quantify it.

By contrast, progress in repeated performances of a skill is pretty easy to quantify, if we set aside the biases associated with assessing it. It may be the case that it is the ready availability of these types of analyses that biases us towards understanding progress purely in terms of developing skills.

So far, I have mentioned two ways of thinking of progress; progress in skills and progress in knowledge. I would now like to introduce a third. Imagine students sit a test at the start of secondary school in English and we then graph their performance in science at the end of secondary school against these initial English results. Assuming invented units of measurement, we are likely to get something that looks a lot like this:

We might conclude that the students in the green loop have made ‘high progress’ and the students in the red loop have made ‘low progress’. But what does this even mean? How does a student make high progress from English to science? What is this telling us?

Whether we can construct a sensible interpretation or not, we can certainly compute some kind of progress score for each student from this data.

Progress is an essential concept if we want to move past the bluntness of raw scores, but it is not an easy concept to pin down. Moreover, the easiest ways of measuring progress may give only a partial picture and may have unintended consequences in the way they distort the curriculum.

An excellent book on this topic is Daisy Chritodoulou’s “Making Good Progress“. Although highly recommended, it still has not resolved all of these issues in my mind.


3 thoughts on “How do we know that our students are making progress?

  1. Tom Burkard says:

    Having spent over half of my working life outside education, I find this discussion a bit surreal: nowhere else do people have the slightest problem devising tests to find out what people have learned or what they can do. In part this is a reflection of the fact that their objectives are not controversial, but rather more significantly, no one’s career is at risk if the examinees haven’t performed well.

    Equally surreal is our insistence on judging pupils by their writing. Some of the best teachers I’ve worked with can’t write, yet somehow they’re expected to teach children how to write effectively. It’s difficult to conceive of a less efficient way of assessing learning–or one more apt to penalise children whose parents and peers don’t speak standard English.

    When the Bew Commission reviewed KS2 SATs in 2011, both Durham’s CEM and our Centre for Policy Studies report recommended the use of Computer Adaptive Tests (CATs), which select from large question banks and use an internal algorithm that selects harder questions in response to a correct answer, and easier ones in response to an incorrect response. CATs are already used extensively in Northern Ireland, the Netherlands and Denmark, and unsurprisingly the CEM also designs CATs. The only way they can be gamed is by blatant cheating–having a ringer sit the test for a low-ability pupil.

    Perhaps more importantly, we should be encouraging teachers to use simple tests and quizzes on a routine basis. There is a considerable body of research demonstrating that it is by far the best means of securing knowledge in long-term memory, and that it improves metacognitive monitoring. After all, I you know you are going to be tested, you pay a lot more attention to what you read or what your teacher says. They also provide teachers with effective feedback. The teachers I know who’ve introduced routine testing have been amazed at the improvement in classroom climate and pupils’ engagement–to say nothing of their GCSEs.

  2. This is timely- we have been charged by ERO (NZs education review office) to accelerate the learning of our year 9s who are behind their peers to a point where they have caught up with the rest by year 11. Ero define acceleration as: “When a student makes more than one year’s progress in a year. A student’s performance is lifted faster than normal as a result of focused teaching. This is done to ‘catch up’ to expected achievement levels. The aim is to have all students on a pathway to achieve at or above National Standard by the end of Year 8, or an appropriate qualification in the secondary context. ” To my mind, the best shot we have at this is to plug the knowledge and reading gap, but this post and daisy’s book will come in very handy. Cheers.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.