PISA is not a test of recall

I have had a number of responses to my posts about PISA from those seeking to dismiss the inquiry learning evidence. This one pithily summarises two of the main arguments that are made:

“That fact that we assume PISA and other tests are still the best measure of learning is troubling. They typically focus on short-term recall and even under the best circumstances, I would argue that if we ask students a year later to take these tests, students, who had more experiential interactions with science would do better.”

Firstly, I am not aware of anyone claiming that PISA is ‘the best measure of learning’. It is a measure of learning and one that seems to provide some interesting results. Would we expect an imperfect measure to produce results that are opposite to those of a perfect measure or would we expect both to point in a similar direction?

I’m not a huge fan of PISA items precisely because they don’t require a large amount of technical knowledge of science. They tend to be tests of application that draw upon broad principles. You can find some of the items here

There is a question on bird migration that describes how birds migrate together in flocks and asks students to select a reason for this. To answer correctly, you need to understand and apply the principle of evolution by natural selection.

There is a question about students investigating the vegetation on different parts of a slope. They then have to answer this open question:

“In investigating the difference in vegetation from one slope to the other, why did the students place two of each instrument on each slope?”

There is a question about meteoroids that assesses whether students can apply their understanding of gravity to identify an explanation for why the meteoroids speed up as they approach the earth.

To answer these questions, students need a broad and flexible understanding of key scientific principles. They cannot answer them by retrieving disconnected facts that they don’t understand. 

The idea that the amount of inquiry learning a student reports correlates negatively with scores on such a test should certainly make advocates of inquiry pause and wonder whether their approach really is the best one for developing flexible knowledge and understanding.

Advertisements

5 Comments on “PISA is not a test of recall”

  1. Mike says:

    …I’m not a huge fan of PISA items precisely because they don’t require a large amount of technical knowledge of science. They tend to be tests of application that draw upon broad principles…

    That was certainly the impression I got from the sample science questions they gave in the SMH the other day – and the other questions in the link you’ve given there tend to confirm it. The one about the craters, for instance, is simple logic and doesn’t require any knowledge of science whatsoever. You do wonder about these tests sometimes!

  2. The questions asked by Eric Mazur in the course of his peer mentoring technique are another example of questions that require the ability to apply knowledge, rather than to recall it (see this example of a question about thermal expansion: https://www.youtube.com/watch?feature=player_detailpage&v=y5qRyf34v3Q#t=1931. What is particularly important, I think, is not just questions that demand the *application* of knowledge but particularly the linking of separate pieces of knowledge.

    In the example I give, 80% of Mazur’s audience gets the question wrong and I don’t think that Mazur really understands why they get it wrong. To answer the question correctly, you need to know not only that metals expand when heated but also that the atoms in a metal are held in a rigid lattice structure, not in a loose conglomeration, like a sponge or bread dough.

    Application therefore frequently requires:
    * the correct diagnosis of the context to which the knowledge is to be applied;
    * the correct identification and recall of the many different items of knowledge that are relevant to the context;
    * understanding how those different pieces of knowledge are related to each other.

    The problem with drill practice is that it treats knowledge as dis-aggregated and isolated items. I worry that proponents of the knowledge-based curriculum do often get stuck on testing such dis-aggregated information. In order to encourage the understanding of the way that information is interlinked and applied, I think we need to develop a better understanding of such structures, and how knowledge is related to the ability to solve more complex tasks. I agree that the PISA questions Greg cites are good examples of that sort of applied knowledge.

  3. Sean says:

    Hi, I am a mature student studying Education as part of a joint honours degree. I have viewed PISA results and been on the OECD website. It seems to me that the PISA test represents a snap shot in time showing the UK’s performance compared to other OWCD member states. From a comparative education perspective it is very hard to draw direct comparisons between the different countries because the way they approach education and their education systems differ so widely. For example in Luxembourg children spend more years in early years learning, start school at a slightly older age, spend less hours in the classroom, do less homework and yet in certain areas of PISA preform better than the UK but unfortunately as comparative education shows what works for one country doesn’t necessarily work for others. I feel one of the down sides to the UK education system is the over prescriptive nature of the NC, means that children are taught in order to pass an exam.

    • Chester Draws says:

      For example in Luxembourg children spend more years in early years learning, start school at a slightly older age, spend less hours in the classroom, do less homework and yet in certain areas of PISA preform better than the UK

      Luxembourg is a small, rich country. Comparing it to the entire UK raises issues completely outside those of comparative education systems.

      If you took the PISA results of inner London (which included all those elite private schools) you would think the UK was stellar.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s