What are you meaningfully changing in your students that cannot be measured?


I had an interesting recent discussion on Twitter. Someone proposed that there are worthwhile educational objectives that are not susceptible to measurement. In other words, we can change students in a meaningful way but we cannot measure whether that change has taken place. I disagree.

Constructs

The kinds of qualities that we may wish to change in students are known as ‘constructs’. If a construct can be observed informally then it can be observed systematically. Systematic observation is what we mean by ‘measurement’. However ideologically distasteful you find the word, that’s essentially what it is.

Constructs range from the ability to add two digit numbers or decode regular words, to less tangible qualities such as creativity. It is obvious that we can observe the former but what about the latter?

If creativity is a meaningful construct, and if it is susceptible to educational interventions, then we should be able to state the observable differences between more and less creative students i.e. we should be able to measure it. If we cannot say what they are, what exactly is the point in making students more ‘creative’?

We may struggle to define these differences, however. In this case, we would need to question whether the construct represents anything meaningful.

A good example might be problem-solving ability. I suspect that relevant knowledge is far more important in solving problems than any general problem solving abilities: Knowledge of algebra helps you solve algebra problems and knowledge of plumbing helps you solve blocked drains.

To assess a general problem solving skill, you would need to assess students on problems where they all lacked relevant knowledge. That would be hard to do. You would also need to keep your assessment secret so that students could not train on the particular problem type. Even then, I suspect you would end up assessing general intelligence – essentially, this is what traditional IQ tests do.

So you could end up proposing that a teaching method will develop problem solving skills but end up testing whether it affects general intelligence. In this case, the problem is not that the construct is unmeasurable, it is that it is a poorly defined construct.

What if unmeasurable constructs do exist?

Even if well-defined and yet unmeasurable constructs did exist, they would not be a good target for educational interventions.

We know that it is possible to intervene with students with the best of intentions, that everyone involved can value the intervention, but that our intervention actually makes things worse.

If the construct is unmeasurable then we could end up doing more harm than good and we would not know. That’s a difficult position to be in ethically, so it might be better to just leave that construct alone.

Clipboards

Does this mean that we must follow our students around with clipboards, constantly recording what they can do? Certainly not. All I am suggesting is that we remove the cop-out of claiming that we are improving some intangible quality in our students. It may make us sleep easier when we receive disappointing maths scores to insist that we are developing far more important abilities that are not susceptible to measurement, but it is not true. Instead, we need to stay humble and pay attention to the evidence as it accumulates, both within our own classrooms and the wider educational community.

Standard

9 thoughts on “What are you meaningfully changing in your students that cannot be measured?

  1. I heard of a school in my region that actually reports to parents on 21st Century type skills – collaboration, creativity, problem-solving. Guess they figured out reliable ways to measure these things and reliable ways to teach them. Far ahead of the rest of us and probably the entire field of educational psychology to boot.

  2. Every school in Victoria reports on those sort of thing: this semester I have to grade my kids’ “critical and creative thinking”. They call it a “capability” and guess what – the vast majority of kids when they have their capabilities assessed fall exactly on the suggested level for their year/age. Fancy that!

  3. panoptical says:

    I teach in the International Baccalaureate Middle Years Programme. My school requires us to report student achievement in the IB’s ten “Approaches to Learning” skill areas, one of which is “Creative Thinking”. The IB gives us a list of specific creative thinking behaviors that can serve as indications that we are teaching students to think creatively. We are encouraged to assess these behaviors with a rubric. They are:

    • Use brainstorming and visual diagrams to generate new ideas and inquiries
    • Consider multiple alternatives, including those that might be unlikely or impossible
    • Create novel solutions to authentic problems
    • Make unexpected or unusual connections between objects and/or ideas
    • Design improvements to existing machines, media and technologies
    • Design new machines, media and technologies
    • Make guesses, ask “what if” questions and generate testable hypotheses
    • Apply existing knowledge to generate new ideas, products or processes
    • Create original works and ideas; use existing works and ideas in new ways
    • Practise flexible thinking—develop multiple opposing, contradictory and complementary arguments
    • Practise visible thinking strategies and techniques
    • Generate metaphors and analogies

    Each learning experience and each assessment that we give students is expected to be aligned with at least one of the behaviors – there are 140 in total across ten categories, including “Critical Thinking”, “Communication”, “Organization,” etc. We don’t have “problem solving” as a general skill, but we do have “Transfer”, which includes things like “Apply skills and knowledge in unfamiliar situations” and “Inquire in different contexts to gain a different perspective.”

  4. I think the problem in this area is that there has been so much focus in recent times on all the “21st-century” guff that we tend to forget the real intangible aims of teaching: instilling in students an appreciation for the subject (and for learning in general), and a desire to further their knowledge independently, and perhaps to add to the store of human knowledge.

    And then there are particular intangible aims for particular subject areas; history teachers will definitely want to instil a genuine sense of historical empathy, language teachers like myself will want to inspire an awareness of the variety of forms of human expression, and the cultural empathy that comes with that. All of these things are intangible and essentially unmeasurable, but they do matter. Otherwise, why would we teach our subjects in the first place? When you think about it, we’d have very little reason to get up in the morning otherwise.

    It’s because the “soft skills” have become such a tick-box exercise that we now expect the intangibles to be measurable somehow. But it’s far better (in my view) to be aware of the distinction and accept that such things as historical empathy and cultural understanding have their place…but that their place is not on a school report.

    • Local school went into Lockdown because nearby one sent a mob over to intimidate parents at the Gates etc. Curiously that part of town has no old fashioned Victorian buildings – every one is 21st century palace. I did a careers talk there (with others, including a professional standup) & I wished someone had called a taxi for me as there was about as much tangible energy in the room as in a Lib-Democrat conference.

  5. Stan says:

    I am always intrigued by why the proponents of problem solving for high school students are not looking at the exemplars in this area. For high school surely one is the Olympiad’s in math, physics, chemistry and so on

    https://en.wikipedia.org/wiki/International_Science_Olympiad

    Perhaps there are good reasons why these have nothing to offer students in general but you’d expect them to at least rate a mention. As in sport the ideas used by those most dedicated to the subject might have some general utility.

    My suggestion anytime you see someone going on about problem solving skills ask them how the ideas from high school science Olympiad’s and the training for that might apply.
    Because what is easily visible here https://artofproblemsolving.com/school/course/woot-physics is it looks like a lot of serious study of the details of the subject and if they are not aware of this then they can be politely asked to go away until they know something about what the best high school problem solvers are doing in math and science.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.