I had an interesting recent discussion on Twitter. Someone proposed that there are worthwhile educational objectives that are not susceptible to measurement. In other words, we can change students in a meaningful way but we cannot measure whether that change has taken place. I disagree.
The kinds of qualities that we may wish to change in students are known as ‘constructs’. If a construct can be observed informally then it can be observed systematically. Systematic observation is what we mean by ‘measurement’. However ideologically distasteful you find the word, that’s essentially what it is.
Constructs range from the ability to add two digit numbers or decode regular words, to less tangible qualities such as creativity. It is obvious that we can observe the former but what about the latter?
If creativity is a meaningful construct, and if it is susceptible to educational interventions, then we should be able to state the observable differences between more and less creative students i.e. we should be able to measure it. If we cannot say what they are, what exactly is the point in making students more ‘creative’?
We may struggle to define these differences, however. In this case, we would need to question whether the construct represents anything meaningful.
A good example might be problem-solving ability. I suspect that relevant knowledge is far more important in solving problems than any general problem solving abilities: Knowledge of algebra helps you solve algebra problems and knowledge of plumbing helps you solve blocked drains.
To assess a general problem solving skill, you would need to assess students on problems where they all lacked relevant knowledge. That would be hard to do. You would also need to keep your assessment secret so that students could not train on the particular problem type. Even then, I suspect you would end up assessing general intelligence – essentially, this is what traditional IQ tests do.
So you could end up proposing that a teaching method will develop problem solving skills but end up testing whether it affects general intelligence. In this case, the problem is not that the construct is unmeasurable, it is that it is a poorly defined construct.
What if unmeasurable constructs do exist?
Even if well-defined and yet unmeasurable constructs did exist, they would not be a good target for educational interventions.
We know that it is possible to intervene with students with the best of intentions, that everyone involved can value the intervention, but that our intervention actually makes things worse.
If the construct is unmeasurable then we could end up doing more harm than good and we would not know. That’s a difficult position to be in ethically, so it might be better to just leave that construct alone.
Does this mean that we must follow our students around with clipboards, constantly recording what they can do? Certainly not. All I am suggesting is that we remove the cop-out of claiming that we are improving some intangible quality in our students. It may make us sleep easier when we receive disappointing maths scores to insist that we are developing far more important abilities that are not susceptible to measurement, but it is not true. Instead, we need to stay humble and pay attention to the evidence as it accumulates, both within our own classrooms and the wider educational community.