Evidence in education

I am unashamedly in favour of using evidence to inform what we do as teachers. This is the reason why I am completing an empirical PhD in an area that I believe to have great potential to guide our practice: Cognitive Load Theory.

However, I also recognise that no matter how good educational research may be, it is never going to tell us how to deal with the boys in the lunch queue. Some commentators will point to this complexity and declare a plague on all educational research. They might even start mentioning ‘critical realism’ or other such verbose and obscure social theories. But there really is no need. My position is a consistent one.

Where there is strong and consistent evidence then we should really be guided by that evidence. Where no such evidence exists then we should use our craft knowledge, underpinned by our humanistic principles. We can do both of these things.

And this is not about Randomised Controlled Trialls (RCTs). These can be incredibly useful but they are generally quite expensive and so there’s not that many to draw upon. They can also be badly designed to the point where they can only deliver a desired result and can tell us little or nothing about the strategies supposedly being tested. 

And it is not about effect sizes. I have become less convinced of this as some kind of across-study comparison measure the more that I have read. An effect size of 0.3 from a well-designed RCT may be far more significant than an effect size of 1.2 from a badly designed, poorly controlled trial with a dodgy test at the end of it. It simply does not all come out in the wash in the way that is sometimes supposed.

Instead, we should be looking for a volume of replicated studies where the strategies that are tested are well-designed. An example of such evidence is the evidence supporting systematic synthetic phonics (SSP) as an approach to teaching reading. Not all of these studies are RCTs. Yet the findings are so consistent that we now have three national panels from the US, UK and Australia all confirming that the weight of evidence supports SSP.

It really would be perverse to ignore it.

Authentic learning

Originally posted elsewhere:

“An exercise is a question that tests the student’s mastery of a narrowly focused technique, usually one that was recently “covered.” Exercises may be hard or easy, but they are never puzzling, for it is always immediately clear how to proceed. Getting the solution may involve hairy technical work, but the path towards solution is always apparent. In contrast, a problem is a question that cannot be answered immediately. Problems are often open-ended, paradoxical, and sometimes unsolvable, and require investigation before one can come close to a solution. Problems and problem solving are at the heart of mathematics. Research mathematicians do nothing but open-ended problem solving. In industry, being able to solve a poorly defined problem is much more important to an employer than being able to, say, invert a matrix. A computer can do the latter, but not the former.”

Paul Zeitz, The Art and Craft of Problem Solving

Imagine a school for surgery. Gone are classes in anatomy or repetitive drill with models. Instead, surgeons are trained in authentic contexts. They learn anatomy just-in-time such as when they are elbow deep in somebody’s abdomen. If this sounds like a bad idea then it is because we readily recognise the difference between an expert surgeon and a surgical student. Those studying surgery need to master a lot of knowledge and skill before they get anywhere near an authentic problem; a real-life human.

In reality, medical students spend a lot of time cutting-up cadavers. They also spend time in lectures and studying from books. A friend of mine had an anatomy chart on the back of the toilet door. And she would memorise it like a parrot. Was this ‘rote’? She certainly memorised things but she also understood them.

Only when such knowledge is robust do students start to integrate it with other skills in an attempt to solve problems. Even the vogue for ‘problem based learning’ for the training of diagnosis starts a good way up the ladder with a huge amount of biological knowledge already assumed.

Unfortunately, this clear distinction between experts and novices fades when life-and-death issues no longer hold it in stark relief. Many assume that the best way to teach mathematics is to require students to behave like professional mathematicians; solving open-ended problems. The best way to learn science is therefore to conduct scientific investigations just like real-life scientists do and, of course, students of history should be ploughing through primary sources.

However, in these contexts, our budding students are likely to learn less. They may become cognitively overloaded, with too much information to process. They may simply be exposed to a problem sub-type so infrequently that they don’t effectively learn from it or they fail to link it to other examples (If you want to improve your golf drive, you don’t play nine holes of golf; you go to the driving range and hit fifty or sixty drives). If you want to train future experts then you need to isolate the various components of that expertise, train those components and then systematically bring them together through a carefully chosen set of contexts that illustrate the deep structure of the concepts or problems.

But there is a persistent clamour for ever more authentic learning. Is your authentic learning not working? Then it must not be authentic enough. And it is a favorite trope of amateurs with an opinion.

It comes from a kind of romanticism. We look at, for instance, mathematical drill in class, we note that this is not always enjoyable for students and we proclaim, “But that’s not what real research mathematicians do!” We yearn for a shortcut so that students may appreciate the beauty of the subject without the grind.

But no such magical techniques exists. I am sure that expert players of the harp derive enormous pleasure from it. Yet, I am not going to replicate that pleasure by just plinking about on a harp myself. I will know that the sounds I make are inexpert and this is unlikely to be motivating. Instead, if I wish to experience the pleasure then I must defer this and undergo the short-term pain of practise – drill, scales and the like.

Maths, science and anything else worth learning are no different. Expertise does not come from simply copying what experts do.

Surf school revisited

Six weeks had passed and Jane from Head Office decided that it was time to return to The City Surf School to see if any progress had been made. A meeting was convened to discuss the action plan.

“So,” said Jane, “I hope you’ve all had a chance to read the research that I sent you about effective surf schools. What are people thinking at this stage?”

There was a pause. Julius and Brian looked to Marie who spoke for the group. “We are explicitly teaching basic skills,” she reported.

Jane smiled with relief, “Oh, that’s excellent!” she said, “so when when did you start with that?”

Marie looked rather stern. “No,” she clarified, “I meant that we are explicitly teaching basic skills. We always have done.”

Jane was now puzzled. “But when I was last here I got the impression that you weren’t keen on the basics; that you saw surf school as being about higher kinds of objectives; learning how to learn water sports and that sort of thing.”

“Then you must not have been really listening,” suggested Julius. “A true dialogue requires all participants to hear.”

Jane wasn’t quite sure what to do with that. Then she had a thought. “So how do you explicitly teach the basic skills of surfing?” she asked.

Marie stiffened a little in her chair, “It is about unpacking the skills for the students; enabling them to make the right connections.”

“Enabling them to make the right connections?” repeated Jane, “That doesn’t sound very explicit to me.”

No-one responded to this comment.

Julius developed Marie’s point. “We assess all the basic strategies that our students use in order to develop their balance. We provide them with the language to talk about the surfing. We give them supports; making the elements of surfing visible to them. We make complex practice accessible.”

“But I don’t know what that means,” complained Jane. “What does that even mean?”

“It really is very clear.” Stated Brian. Everyone briefly looked at Brian.

“OK,” said Jane, “Do you practice these skills on the sand before going in to the water?”

Julius bristled. Disgust transited his face. There was a pause.

Marie responded, “We unpack the skills for the students.”

“On the sand?” Asked Jane.

“We often engage in instruction on the sand,” Marie suggested.

“Yes,” agreed Jane, “but do you get your students to crouch on a board that is on the sand and then stand up on that board in order to practice the manoeuvre that they will need to do out in the sea?”

Quietly and slowly, Julius hissed, “We teach in an authentic, situated and engaging way.”

Brian nodded. “I have a motivational poster,” he added.

Jane was struggling to hide her frustration. “But successful surf schools directly teach the skills necessary for surfing. All the evidence says that you have to explicitly teach these basic skills and get the students to practice them!”

“I know,” said Marie, “I could have told you that. That’s exactly what we do.”

[The original surf school post may be found here]

Let us be explicit about the explicit

I often write about ‘explicit teaching’ and I have chosen this term with great care. The alternative term, ‘direct instruction’ is problematic due to the fact that it has several, closely-related meanings. Conventionally, ‘direct instruction’ with lower-case first letters tends to have the same meaning as explicit instruction and this approach is summarised well by Barak Rosenshine here (he did much of the initial process-product research that led to a description of this teaching method).

However, ‘Direct Instruction’ with capitalised first letters usually refers to specific programs develop by, or using the methods of, Siegfried Engelmann and his associates. This includes all of the features of explicit teaching but adds a curriculum design element; specifically, lessons and units are planned by specialist planners and the lessons themselves are effectively scripted. It is to avoid confusing these two meanings of ‘direct instruction’ that I choose to use the term ‘explicit teaching’.

It was therefore with horror that I encountered (via @BarryGarelick) a presentation by a highly influential teacher educator – Deborah Loewenberg Ball – to the National Council of Teachers of Maths (NCTM) in the US. In this presentation we are informed that explicit teaching is not direct instruction:

Hmmm.. It seems to me that explicit instruction therefore differs from direct instruction in that it leaves more items implicit. In direct instruction, presentations show students how to complete tasks but, presumably, in explicit teaching, they do not.

I certainly do not agree with the view that all tasks completed through direct instruction are ‘uncomplicated’. The purpose of breaking tasks down into their constituent parts is in order to not overload working memory. This way, in incremental steps we can develop students to the point of completing really very complicated tasks. The idea is that we just don’t dump them in there from the outset and let them fend for themselves, like in the Hunger Games or something.

It also should be made clear that this is a very personal interpretation of what ‘explicit teaching’ means and that it is one that is not widely shared. For instance, the Centre for Education Statistics and Evaluation in New South Wales (CESE) promotes explicit teaching and gives a definition much more in line with direct instruction:

The key problem here is that there is significant potential for confusion. Teachers could be left unaware of where the evidence lies. Researchers, of course, have the right to promote their views and their research. However, if researchers employ this eccentric definition of ‘explicit teaching’, teachers might mistakenly think that all of the research supporting explicit teaching practices, such as the evidence Rosenshine cites above, supports the practices that Deborah Loewenberg Ball is advocating.

That would be unfortunate. The research supports direct instruction.

Hey Google, what does 'explicit' mean?

Hey Google, what does ‘explicit’ mean?

Mandating Bad Ideas (a persistent problem in education)

Inquiry learning may have a place in the curriculum. It can give students an idea of the processes of a discipline and it can be motivating (it can also be excruciatingly dull). However, it is less effective than explicit instruction for teaching concepts and procedures to subject area novices; the sorts of students who are learning academic subjects in K-12 education.

Not everyone agrees with this position and that’s fine. It is an issue about which reasonable people may disagree. However, I suspect that few of those who disagree with me would be likely to support the idea of mandating inquiry learning in schools across a whole region.

And yet this is what is currently happening in Victoria.

Students in Victoria who wish to go on to university typically study Victorian Certificate of Education (VCE) subjects in Years 11 and 12. Year 11 is entirely school-assessed and all that needs to be reported to the Victorian Curriculum and Assessment Authority (VCAA) is whether students have ‘satisfactorily completed’ particular subjects.

In Year 12, VCE subjects are assessed by coursework and a final exam. I actually like this system because the coursework is moderated against a school’s exam grades, meaning that all that really matters is how the school ranks students from the coursework. For instance, if a school obtains five A+ grades in the final physics exam then the top five ranked students will be given A+ grades for their physics coursework. The beauty of this system is that it prevents some of the dodgy practices that sometimes arise from school-based assessment.

The study designs (syllabuses) for mathematics and science subjects have been revised for the start of 2016 and this seems to have been an opportunity for some to pursue an inquiry learning agenda.

In Mathematical Methods, the coursework for Unit 3 (half of the Year 12 course) currently consists of an ‘application task’ – effectively an extended, contextualised question – and two tests. In the new study design, these have all been replaced by a single ‘application task’ that has been redefined to be:

A function and calculus-based mathematical investigation of a practical or theoretical context involving content from two or more areas of study, with the following three components of increasing complexity:

• introduction of the context through specific cases or examples
• consideration of general features of the context
• variation or further specification of assumption or conditions involved in the context to focus on a particular feature or aspect related to the context.

The application task is to be of 4–6 hours duration over a period of 1–2 weeks.”

So it is an investigation and it has to last at least 4 hours. When I attended a recent briefing, I was told that it should not be run under exam conditions and that we were encouraged to allow students to ‘collaborate’.  Exactly how it is possible to fairly assess each individual under such conditions is unclear. The task should also apparently run in one continuous block – we can’t break it up into separate chunks and run them at different times.

This is uncannily similar to what has happened in the new physics study design. To be fair, the current physics study design requires students to carry out an extended practical investigation but considerable scope is given to schools as to how to organise this. I am not against practical work; I like the model of a practical exam that we used to have in England. The VCE requirement that students have to design the investigation themselves, although completely misguided given their level of knowledge and understanding, is something that I have been able to work with due to the considerable scope schools have for interpreting the guidance.

However, the VCAA must have concluded that schools were using this freedom to not conduct a pure enough kind of investigation. So, we see more prescriptive guidance in the new study design, complete with a burdensome time requirement. Students have to make a scientific poster according to the VCAA’s own template that must not exceed 1000 words and they must also devote at least seven hours to this investigation and its write-up.

These developments are no accident. I was at a couple of the initial ‘consultations’ when these study designs were being revised and a lot of the discourse was framed around how to incorporate more inquiry learning into the physics and chemistry VCEs. The goodness of inquiry learning wasn’t in question, just the mechanics of how to shove more of it in.

I think the time is ripe for disruption of qualifications in Australia. There was a telling moment at the briefing about the new maths study design when one teacher asked the presenter if universities had been consulted. The presenter quickly answered, “No.” Would you design a system of academic exams that did not consult with universities on their content?

I suspect, if asked, the universities would prefer a far more conceptual course, less dependent on expensive calculators where, instead, students focus on really developing their algebra and calculus. I don’t think that the International Baccalaureate is the answer – although I’ve not taught it – and so perhaps there is an argument for a new Australian exam board to compete against the current offerings.

It’s all a bit too cosy at the moment.

Teaching for Understanding

We all want our students to understand what they learn, right? I can’t think of any teacher that does not. However, the idea of teaching for understanding can be problematic and lead us into some serious, and highly popular, misconceptions.

In this post, I will outline what understanding actually is before explaining the best way that we can promote it.

Rote learning

Many teachers are keen to avoid ‘rote’ learning. This is when students memorise facts and procedures without really understanding why they are doing it. It is possible to do this, especially if you practice something a lot like a mathematical procedure. However, memorising facts that have no meaning to you is extraordinarily difficult. This is why memory champions who can memorise the order of a deck of cards, for instance, tend to impose a spurious meaning on it. They may imagine walking through a house and seeing a different card in each room.

We also do this in education. At school, I learnt the order of the colours of the rainbow by memorising, “Richard Of York Gave Battle In Vain.” Unfortunately, I wasn’t taught history properly and so the resonances that the writer of this mnemonic intended were rather lost on me.

So, by repetition, you can learn the ‘how to’ of something like a mathematical procedure without learning the ‘why’. And you can also learn key facts this way such as times-tables. However, anyone genuinely attempting to teach an entire curriculum in a rote fashion would find the task impossible. Those who maintain that standardised tests are largely tests of rote learning are likely wrong.

The nature of understanding

I think that one problem arises when we start to see understanding as something qualitatively different to knowledge. Don’t mistake me here, understanding is a useful concept and I am not trying to replace it but we need to recognise that it is the degree of a thing rather than a thing in itself. In their influential book “Understanding by Design,” Grant Wiggins and Jay McTighe suggest that pieces of knowledge are like tiles and understanding is like the pattern that the tiles make. This is useful as far as it goes and brings to mind the psychological concept of a ‘schema’ where related ideas are organised in the mind and therefore understanding represents this organisation. However, I suggest that a fractal is a more apt analogy; it’s tiles all the way down.

I have written before of two types of understanding; objective and subjective. Objective understanding is where we perceive that someone else understands a concept. In his book, “Intuition Pumps And Other Tools For Thinking,” the philosopher Daniel Dennett contrasts a small child saying, ‘my daddy is a doctor,’ with a young adult saying, ‘my dad is a doctor’. He draws this comparison for different reasons to me but I would point out that we would perceive the small child to understand this statement less than the young adult. What accounts for the difference? Well the young adult will have a lot more relevant world knowledge; she will know that doctors work in hospitals or clinics, that there are different kinds of doctors, that doctors are relatively well paid and so on. So her deeper understanding actually consists of a larger amount of relevant knowledge.

Subjective understanding is the feeling that you understand something. This can be quite deceptive. However, assuming that the feeling is accurate, what accounts for it? Well, let’s look at the opposite; a feeling of confusion. This can be easily induced by presenting a student with a complex problem with many elements to attend to. However, if the student recognises many of the elements because she already has ideas about them available in long term memory then the confusion will be diminished. Sufficient relevant knowledge will mean that the student feels that she understands the problem. Again, understanding is effectively an accumulation of relevant knowledge.

Further, this can be facilitated by some procedural knowledge. If you can automatically perform the ‘how to’ part of a mathematical problem or immediately recall the answer to “6 times 8” then you don’t have to devote attention to these parts of the problem and will have a better chance at apprehending the ‘why’. As Alfred North Whitehead suggests:

“It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.”

Dangerous Knowledge

Unfortunately, as Whitehead suggests, many people draw precisely the opposite conclusion about understanding. In their famous study, Kamii and Dominick conclude that directly teaching mathematical procedures actually gets in the way of students understanding the maths. These studies are not perfectly controlled experiments and an attempted replication (of sorts) by Stephen Norton in Queensland found the opposite result.

This is also conventional wisdom in science education. Teaching ‘facts’ is harmful to understanding and should be replaced by students engaged in activity and finding things out for themselves.

The logic seems to be:

1. Traditional approaches to teaching leave students with lots of misconceptions

2. ‘Constructivist’ teaching approaches are an alternative to traditional approaches

3. Therefore, we must use constructivist teaching approaches

This is a classic non sequitur. How do we know that the constructivist approaches will work any better? The evidence in favour of them is extremely thin. It follows the logic of the politician’s syllogism:

1. We must do something

2. This is something

3. Therefore, we must do this


Unfortunately, this sort of thing sends us off on a wild-goose chase. Instead, we should be looking at how we can improve conventional teaching approaches so that we can reduce misconceptions and maximise students’ levels of understanding. Interestingly, an important difference between experts and novices appears to be the ability to perceive the ‘deep structure’ of problems and situations. There are two key strategies to consider if we want to develop this appreciation of deep structure.

Woefully neglected in the research, teacher explanations are pretty critical to developing student understanding. I wrote about this for the TES. Unfortunately, it is not clear exactly how we can structure explanations in order to make them optimal. We can perhaps infer a few principles from the second strategy.

Building episodic knowledge seems to be effective – ‘episodic knowledge’ just means students’ memory of encountering similar problems or situations. This involves running students through many examples with the same deep structure. In physics, this might mean teaching a concept or principle, giving an example, asking students to complete similar examples before gradually and progressively diversifying into other problems with the same deep structure but different surface features. Once a number of different principles have been grasped then students can be presented with problems with different deep structures to see if they can identify what this is in each problem.

This suggests also that explanations should not just focus on the principles but also highlight the deep structure in different problems and situations, perhaps comparing these situations. In mathematics, you may wish to move through multiple representations e.g. from a rule to a graph and back again. The point is that the deep structure is retained between these representations whilst the surface features change, again allowing students to better grasp this structure.

We can’t bypass this stage. It appears that it is quite natural for new learning to be locked to surface features initially and it takes a lot of hard work to move past this.

We don’t have to choose between knowledge and understanding

It is both misconceived and potentially harmful to conceive of understanding as a spooky kind of a thing that is qualitatively different to knowledge. It leads us into thinking that gaining knowledge interferes with gaining understanding whereas the reverse is the case; the two are mutually supportive. And it leads us away from more effective approaches for developing understanding.

The false choice at the heart of Freire

Whenever I write about Freire’s book, “Pedagogy of the Oppressed”, I tend to provoke one of three reactions:

1. That I am reading the wrong Freire book and I should read other ones instead

2. That I don’t really understand what Freire means (this usually comes after I have quoted him)

3. That nobody is really influenced by Freire anyway

I have to admit that it is a very odd book. It is mainly about revolution and is written almost entirely in the abstract about the ‘oppressed’ and the ‘oppressors’. It is hard to even imagine what this means. However, I think that this acts as something of a blank canvas onto which we may can project our own preoccupations. I might imagine peasants versus dictators; you might imagine students versus teachers. If I ask a student to remove his coat in class then perhaps I am the oppressor and he is the oppressed? What would Freire say?

Indeed, something has to account for the book’s extraordinary influence. According to the Pedagogy of the Oppressed website,

“Over one million copies of Pedagogy of the Oppressed have been sold worldwide since the first English translation in 1970. It has been used on courses as varied as Philosophy of Education, Liberation Theology, Introduction to Marxism, Critical Issues in Contemporary Education, Communication Ethics and Education Policy.”

“Pedagogy of the Oppressed is one of the foundational texts in the field of critical pedagogy, which attempts to help students question and challenge domination, and the beliefs and practices that dominate.”

And so it sits at the base of a whole field; critical pedagogy.

At a more trivial level, Pedagogy of the Oppressed is a source of internet memes and it was a recent encounter with a tweeter of such memes that started me thinking again about Freire. The source of these memes is generally chapter 2 where Freire has a go at what he calls the ‘banking concept’ of education where teachers ‘deposit’ knowledge in students and which sounds to me a lot like explicit teaching. I think this is one of the reasons for the book’s huge popularity. People can use it as a source of authority from which to criticise explicit forms of instruction.

The book is laced with ironies. For instance, Freire criticises traditional teaching for setting up dichotomies, as encapsulated by this (misspelt) meme:

banking concept II

There’s a lot of wild assertions there (necrophily? seriously? that’s just weird…) and the book itself does little to substantiate them. However, let’s stick with this idea of dichotomies.

A false dichotomy is when a someone presents us with two options when, in reality, there are more. For example, if I said, “You either oppose standardised testing or you are a neoliberal,” then that would be a false choice. Standardised NAPLAN testing was introduced to Australia by a left-of-centre government. Indeed, the sort of system where the state sets targets and then measures progress against them is perfectly consistent with state socialism. The point is that the choice presented in the statement does not cover all of the options. You can see why this is a bad thing*. Going around dichotomising everything would be problematic because it would risk creating false choices and so risk grossly oversimplifying the world.

Yet here is another Freire meme.

banking concept


So, what is the alternative to the banking model? According to Freire, it is ‘problem-posing’ education. If this sounds familiar, it is because there is a problem-based theme running from at least William Kilpatrick in 1918, through Freire and right up to today’s proponents of inquiry learning, the maker movement and project/problem based learning. Freire’s problem-posing education seemed to consist of showing images to peasants and trying to initiate a dialogue around those images. Yet even then, it’s not supposed to be the teacher who selects these images in isolation – they should draw upon what the peasants wanted to investigate.

So, a teacher must either continue using the banking model where, “by considering [students’] ignorance absolute, he justifies his own existence,” or he can engage in some sort of problem-posing education. But this doesn’t cover all of the available options, does it?

When I teach Year 12 physics, I am well aware that my students already know quite a lot of physics. In fact, I hope they do and my starting point is always to find out exactly what. I assume that they know a lot of other things too. Analogies would hold no explanatory value if my students didn’t know about the thing that I was using as an analogy. Yet, I don’t do anything like problem-posing teaching. I stand, usually at the front, explaining things to the students and asking them questions before setting them tasks to complete. This is Freire’s banking concept.

So Freire himself has set up a false choice.

Now, it is possible that we’ve over-extended Freire here. He was writing about a particular form of education – the education of illiterate adult peasants – at a particular time. I happen to think that these peasants might have been better served by being taught to read in a systematic way but let’s set that aside for now.

I would be happy enough to agree that Freire has little of value to say about educating students in the today’s schools. If so, we would probably have to conclude that Critical Pedagogy, at least in its application to schools, is built on a false premise.

*Unfortunately, people tend to be a little too eager and look for false choices in a way that sometimes limits debate. For instance, at the moment that a teacher first teaches a new concept to her students then she can either fully explain that concept or rely on some degree of discovery on the part of the students. There are no other options available and so discussing the relative merits of these approaches is not setting up a false choice.