Obtaining a high success rate

“To learn efficiently, students must be engaged in activities that are appropriate in level of difficulty and otherwise suited to their current achievement levels and needs. It is important not only to maximize content coverage by pacing the students briskly through the curriculum, but also to see that they make continuous progress all along the way, moving through small steps with high (or at least moderate) rates of success and minimal confusion or frustration. If lessons are to run smoothly without loss of momentum and students are to work on assignments with high levels of success, teachers must be effective in diagnosing learning needs and prescribing appropriate activities.Their questions must usually (about 75% of the time) yield correct answers and seldom yield no response at all; their seat work activities must be completed with 90-100% success by most students.”

Teacher Behavior and Student Achievement, Jere Brophy and Thomas Good, April 1984

One of the key findings of the process-product education research of the 1950s-1970s was that more effective teachers seek and obtain a high success rate from their students. In a sense, this may seem like a tautology. Success is an outcome measure associated with effective teaching. So it’s a little like suggesting a comedian be funny or a runner be fast. Nonetheless, it’s not hard to see why such a factor would be of interest to researchers – nothing is easier to observe and quantify in a classroom than the proportion of questions that students answer correctly.

The question arises as to whether this is of any consequence to teachers. Despite the potential circular logic, I do think the exhortation to obtain a high success rate is useful, with one or two caveats. First, I will explain what I think a bad interpretation of this would be and then I will explain how I think it can be of use.

One conclusion we may draw from the need to obtain a high success rate is that we should reduce the demand of the curriculum. Instead of asking students to grapple with Pythagoras’ theorem, we could get them to do an investigation where they measure the lengths of the sides of various triangle and make a poster about this. I do not think this is a useful way to interpret this research finding because there is no strategy for meeting important goals. We want to enable students to jump over the bar where it sits, not permanently lower the bar.

So, instead, I would suggest two ways of obtaining a high success rate that are not entirely distinct. The first is a building-up strategy. To extend the metaphor, when we are building up, we gradually increase the height of the bar, providing feedback and reteaching at any point where students cannot clear it. Eventually, they are able to clear the high bar. In scaffolding, we keep the bar where it is but provide lifts and supports that we gradually take away until the student can clear it for themselves. The role of formative assessment is obvious because, without it, we cannot work out whether the bar has been cleared or not.

Building-up is classic explicit instruction. By November, my Year 12 mathematics students will need to be able to answer some pretty complex probability questions. The process of building up to this began way back in primary school with simple probability concepts. They are now at the stage where they can draw tree diagrams and have used the various probability relationships in a narrow range of contexts, but they are not yet generally able to answer the more advanced questions that they will face at the end of the year.

So part of the space between now and then involves building up through simpler examples to more complex ones.

However, I have also started scaffolding. This is when I take a full, complex question and structure it in such a way that I can obtain a high success rate even at this early point.

Here is an example. First, I project the opening blurb of a problem before any specific questions have been asked.

The probability distribution of a discrete random variable, X, is given by the table below.

x 0 1 2 3 4
Pr(X=x) 0.2 0.6p2 0.1 1-p 0.1

At this point, I ask students to show me, on their mini-whiteboards, the value that the probabilities must sum to. This is not something that is asked in the question but it is something the students need to know in order to be able to answer it. If I obtain a high success rate here, I can move on to ask them what are the allowable values of p. Here, I stress the important point that 0.6p2 and 1-p must be numbers between 0 and 1, including 0 and 1. I ask them to annotate their copy of the question with these two conditions.

I then project the first part of the question:

a. Find p

At this point, I would ask the students to sum the probabilities, set them equal to one and collect like terms in order to get a quadratic equation. They should be able to do this. I would then ask how to deal with the quadratic equation, given that it contains of decimals. What strategy can we use? I would want to see ‘x 10’ or, better still ‘x 5’ on the mini-whiteboard. We would then discuss this.

Finally, I would ask them to factorise the quadratic equation and solve for p. The answers should be p=2/3 or p=1. I would then ask them to test that both answers satisfy the condition that 0.6p2 and 1-p must be numbers between 0 and 1, including 0 and 1. In this case, they do.

This way, a question that far fewer than 75% of them would have been able to answer cold, will likely have been answered correctly by virtually all of the students.

It’s easy to see why. I have stopped students from going off in the wrong direction. I have reminded them of relevant steps. In this case, both of the calculated answers are allowed, but I have reminded students that they need to consider this point in such questions. In terms of motivation, I have given them a sense of success, enabling them to tackle a question they would have previously found daunting.

Ultimately, ‘obtain a high success rate’ comes from an observation of what effective teachers do. But it is not entirely tautological. I think there is a message in there for us.

An apology

The truth is that not everyone on social media is particularly pleasant. I am sure they are fine in real life, but Twitter can bring out the worst in people.

When I first joined the medium, I tended to put up with this. The trouble is that as you gain followers, ever more of the bad stuff is flung at you and so I am now pretty quick to mute or block people who resort to personal attacks.

Although this stops me from seeing this stuff, it is unfortunately of little help to you. If you tweet out one of my blog posts or reply to me on Twitter these people are likely to haunt your timeline, telling stories about me. I am sorry about that.

The stories are virtually always untrue. I do not block people for disagreeing with me. I am not a misogynist/racist/[insert here]ist, I am not on the ‘far right’* and I do not incite my followers to attack people. If you are one of my followers, you will know that.

My suggestion is to ignore these people or get rid of them from your timeline. If you have the patience, ask to see the evidence to support their accusations. There won’t be any.

*It is a source of great sorrow to me that I have been on the centre-left all of my life and yet such accusations are still made. I have only ever voted for the British Labour Party and the Australian Labor Party but in these polarised times, pretty much anyone can be accused of being far-right. This is not just a personal complaint. The net effect of this phenomenon is to destroy the centre-left, making the left unelectable in the process and handing control of the world to right-wing populists like Donald Trump.

Structured Word Inquiry gets curiouser and curiouser

One question that baffles many of us about Structured Word Inquiry (SWI) is how it could possibly be used as a beginning reading program. In a previous post, I mentioned Peter Bowers’ video that was intended to demonstrate just how this is done but where the child could apparently already read, “Mom says she wants a…”. So it was with interest that my attention was drawn to an initial reading program developed by an SWI proponent.

I have to say that it starts oddly with an address that is ostensibly directed towards the person who is about to learn to read:

“Congratulations! You’re going to learn how to read! But, before you read a single word, you need to know some facts about the English spelling system… A grapheme is the smallest meaningful contrastive unit in a writing system. <h>, <sh>, and <ugh> are graphemes. Single letters can be graphemes. Graphemes can be made of two letters. A two-letter grapheme is called a digraph. The prefix di- means “two.” Graphemes can be made of three letters. A three-letter grapheme is called a trigraph. The prefix tri- means “three.”.”

And so on. I assume that we are not to take this literally and that it is not being suggested that we talk to illiterate five-year-olds about the ‘smallest meaningful contrastive unit’ or later that, ‘Letters can function as lexical, phonological, and/or etymological markers’. Instead, I will assume that this is a colourful way of informing the instructor about some of these concepts.

However, I do think this preamble is revealing. At one point, we are told that, “English spelling is rule-based. There are no exceptions, just more rules to uncover.” Then later on in the text, there is a discussion of homophones:

“Homophones provide further evidence that the primary function of English orthography is to represent meaning. Homophones are two or more words that sound the same but that have different spellings and meanings. The homophone principle states that, when two words sound the same but have different meanings, the spellings will be different whenever possible.” [my emphasis].

That ‘whenever possible’ is carrying a lot of weight here because there are indeed words known as ‘homonyms‘ that sound the same, are spelled the same way but that have different meanings. We may have been worried about these being exceptions to the rule that ‘when two words sound the same but have different meanings, the spellings will be different’ but no. In the case of homonyms, presumably this is not possible. All is well with the world. Order is restored.

I am reminded first of the epicycles that accounted for the motion of planets in a universe with the Earth at its centre and all heavenly objects in orbit around it. I am also reminded of Godel’s theorem. But more of that later.

It seems to be important to proponents of SWI that English spelling is rule-based with no exceptions because a key criticism they make of phonics seems to be its use of rules that have exceptions. Here are Jeffrey and Peter Bowers writing together on the subject:

“It is widely assumed that the primary role of letters in English is to represent sounds, and the many “exception” words are generally taken to reflect a poorly designed spelling system. However, this reflects a misunderstanding. In fact, the English spelling system is designed to encode both pronunciation and meaning of words, and as a consequence, English word spellings are constrained by phonology, morphology, and etymology. Rather than a perverse  system that needs reform, some linguists call English spelling “near optimal” (Chomsky & Halle, 1968). Whether or not English spellings are near optimal, the key claim we would make is that English spellings are logical and can be investigated like other scientific subjects given that the English spelling system is systematic.”

Repeat after me the Orwellian mantra: There are no exceptions, just more rules to uncover.

The SWI argument strikes me as peculiar because it appears to derive the logic of the English language from written English. It is plain as day that written English has had a profound effect on spoken English, but it is also clear that spoken English has primacy. If we all start saying something new then written English has to accommodate it. There is no process by which written English can lodge a formal objection to innovations in spoken English.

For example, take the word, ‘slippy’. It clearly exists because you can look it up in a dictionary. At some point, someone must have decided that ‘slippery’ was too much of a mouthful, they couldn’t be bothered with it and so they would say ‘slippy’ instead. But that’s just my own speculation because when I tried to look it up in the online etymology dictionary, I couldn’t find it. So I’ve no idea what rules govern the spelling of this word. They must be yet to be uncovered because, remember, there are no exceptions. None. Zilch. Nada.

Holding such an absolutist stance seems like a fundamental flaw. Just one bona fide exception and the SWI edifice crumbles. There is no room for pragmatism here or for a shrug of the shoulders and a wry smile at the eccentricities of English. The inductive logic of science is replaced by the deductive logic of mathematics. This is probably why SWI proponents spend so much time on Twitter trying to dig themselves out of holes that their all-encompassing generalisations have thrown them into.

And there is an interesting warning from mathematics – a warning for anyone who wishes to construct complete formal systems of logic, epicycles and all. In 1931, Godel published his mathematical demonstration that no formal system of mathematical axioms can prove all truths about the arithmetic of natural numbers. In other words, even in mathematics, whatever system of logic you decide upon, there will be exceptions.

Maybe the logic of English spelling will turn out to be more robust than the logic of basic mathematics. Time will tell.

In the meantime, repeat after me: There are no exceptions, just more rules to uncover.

Teaching children how to think etc.

Another decade, another article in The Conversation illustrating the irony that advocates of critical thinking seldom think critically about it.

There are two interlinked but distinct concepts that we have to consider about critical thinking. Firstly, how general are these skills and, secondly, how transferable are they? They may sound like the same thing, but they are not. And they both have real, practical consequences.

Firstly, let’s consider generality. Those who tend to buy into the view that critical thinking is a general skill (or a ‘general capability’ as it is termed in the Australian Curriculum) can fall into two main traps. The first is to view the contexts for thinking critically as interchangeable. This is what we see when science lessons become projects involving cardboard boxes and LEDs. The precise science covered is not considered important because students are developing their skills of creativity, critical thinking, problem solving and so on. However, if you view these skills (or ‘virtues’ as Carl Bereiter calls them) as being highly domain specific, then these students have merely developed the ability to think critically, creatively and so on in the domain of cardboard boxes and LEDs. The value of this to the student then becomes debatable, especially when contrasted with actually learning some science.

The second trap is to think that, because these skills are general, you can teach a discrete critical thinking course or bolt a critical thinking module onto the curriculum that deals with these strategies in the abstract or in model contexts.

When we start to consider the kinds of strategies that may be more generally applicable, we often alight on maxims such as ‘look at the problem from different perspectives’ or the kinds of rules-of-thumb embedded in logical fallacies. I tend to think that there is a trade-off. The more widely applicable such a strategy or rule-of-thumb is then the less useful it tends to be. One example is perhaps the maxim that, ‘correlation is not causation’:

This maxim is applicable to a number of fields where we seek correlational evidence, from the sciences, through social science and into the humanities. There are even philosophers who will go on about it a bit. And you may think it has high utility. But let’s imagine you wade into a discussion between two public health officials about the health impacts of smoking with your pithy observation that ‘correlation is not causation’. You are likely to get short shrift. Why? Because they have a lot more contextual, domain-relevant knowledge. They can probably discuss the extent of the correlational evidence and the tipping point that signifies acceptance of a causal relationship. If you try to apply this maxim when reading an article written by one of these officials, it may lead you into error.

Now consider someone making claims that school exclusions cause knife crime based upon statistics showing rising levels of both. Here, the maxim that ‘correlation is not causation’ may point you in the right direction. It is plausible, for instance, that the same thing that causes students to be excluded from school is causing the knife crime. Or that knife crime causes students to be excluded from school. But how can you tell the difference? Armed only with your maxim, how will you know when to apply it and when not to apply it? The only way you can figure that out is by learning lots about the evidence behind smoking or knife crime. In other words, the way you ultimately establish a reasoned position is by learning about the domain in question. The maxim alone could lead you as often into error as to the truth.

So what of transfer?

Even if we accept that ‘correlation is not causation’ has some general utility, it is not certain that people will apply it generally. Odd as it sounds, there is quite a lot of evidence that when people learn strategies that do work across different domains, the strategy somehow becomes locked to the domain in which they learnt it and so people fail to apply it to other situations where it would also add value. Dan Willingham highlights this in his discussion of two problems that involve a tumour and a fortress.

Latterly, proponents of critical thinking appear to have become more aware of the common criticisms and will make statements such as, ‘of course, knowledge is important to critical thinking’. However, you cannot have it both ways. If critical thinking just represent the highest levels of performance within a traditional subject discipline, then we do not need critical thinking courses or a special focus on critical thinking. We just need to teach our subjects really well. However, if we believe that it deserves special treatment, then why do we think that, unless it has a more general impact?

Peter Ellerton, the author of the piece in The Conversation, proposes that we consider the example of Philosophy for Children, a, ‘program that involves teaching the methodology of argument and focuses on thinking skills.’ So it is a discrete critical thinking course. Ellerton also thinks it will have a general impact because he states that, ‘Studies involving a Philosophy for Children approach show children experience cognitive gains, as measured by improved academic outcomes, for several years after having weekly classes for a year compared to their peers.’ In other words, deliver Philosophy for Children and you will produce gains in wider academic outcomes. I am not convinced by the study Ellerton cites. It is not randomised and there is high attrition in the experimental versus control groups (28% versus 9%).

A perhaps more thorough study of the benefits of Philosophy for Children is the Education Endowment Foundation randomised controlled trial that found that children who had the intervention made greater gains in reading than those who did not. However, this was an odd trial which I have discussed at great length on this blog. The scale-up study is scheduled to report early next year and then we will see more evidence supporting or refuting the extraordinary idea that discussing whether it is OK to hit your teddy bear impacts on reading ability.

Until then, proponents of critical thinking, although they may be welcomed at conferences, have a credibility gap. They need to demonstrate two things, firstly that these skills are generally useful and, secondly, that students will transfer their learning of these skills to the various different domains where they apply. Until then, we should ask, as Carl Bereiter does:

“Nobel Laureates, captains of industry, cabinet ministers, school superintendents – any one of them is likely to end a commencement address or a discourse on the current crisis by declaring that schools have got to ‘teach students how to think’. The words roll easily off the tongue and the speakers show not the slightest doubt that the words mean something. But do they?”

The golden rule of EduTwitter

I’ve been active on Twitter now for nearly eight years, mainly discussing education. EduTwitter is a funny place. People tend to go into the field of education for the altruistic reason of helping children and making a better future. Which is definitely a good thing. The danger arises because we then tend to assume that criticism can only come from those who dislike children and don’t care about the future.

There are a number of fallacies that tend to derail EduTwitter debates. A fairly comprehensive list may be found here. I also repeatedly point to Paul Graham’s article on how to disagree which neatly and clearly explains some of the most common traps. However, I now think that one principle, above all, can help us avoid false reasoning. When I look at the people I have blocked, they have invariably fallen foul of this principle and so I believe that violating it leads to the most negative experiences on the platform.

I tweeted the principle yesterday:

The key is to accept a dualism – that there are ideas and then there are the people who have those ideas. It is legitimate to attack the former but not the latter. Attacking the person and not the argument is ‘ad hominem‘ and it is by far the most frequent fallacy I encounter, one that is uniquely unpleasant.

Of course, I understand that people become upset and offended when their ideas are criticised. That happens to me too. And it is possible to be on the end of an ad hominem argument and just shrug your shoulders, such as when people point out that I am not an early literacy expert. I’m not an early literacy expert and I have nothing invested in the idea that I am. I am also amused when people advise me to focus on finishing my PhD.

However, if I become personally upset because someone challenges my argument then the fault is with me and not really with the person making the argument. And even if I care little about an ad hominem attack, it pretty much always signals the kind of bad argument we would all be better off without.

Even so, why should we allow for ‘mockery’ of ideas? Why do we need that? Doesn’t it make the place less pleasant?

There are two reasons why it is absolutely essential that we allow mockery of ideas. The first, as Paul Graham points out, is that tone is almost impossible to read on the internet because we lack the physical and auditory cues that gives this away. Politeness is easily mistaken for sarcasm, particularly if you are not well inclined towards the argument of the person who is being polite. Prohibiting mockery gives license to endless tone wars that divert us from the substance of what anyone is attempting to claim.

Secondly, humour is a critical tool for pointing out the absurdity of some arguments. It is no wonder that tyrants and ideologues through the ages have tried to limit our ability to make jokes.

So I think the main point of focus should be ensuring our arguments, whatever they are, are targeted at ideas and not people. I have not always managed to do this myself, but I will redouble my efforts in future.

Structured Word Inquiry fails a key test

Following yesterday’s post on Structured Word Inquiry (SWI), Kathy Rastle highlighted in the comments that The Nuffield Foundation have conducted a randomised controlled trial comparing SWI with something called ‘motivated reading’ as interventions for struggling readers.

An intervention for struggling readers is not the same as initial reading instruction but interventions are often the subject of research because they tend to be more contained and manipulable. In this case, there appears to be no published paper on the study, but the research team have released a set of slides destining the study and its findings.

Critically, this was a developer-led study. In other words, SWI developers were involved at all stages, unlike a program out in the wild.

Interestingly, the SWI researchers refer to it as the ‘MORPH’ project, illustrate the slides with images of a cartoon character called ‘Morph’ and refer to SWI as a form of ‘morphological instruction’. I have been called-out by SWI advocates on Twitter for suggesting that SWI focuses on teaching morphology from the start and I am now beginning to wonder why.

That phrase ‘from the start’ is important. The slides contain examples of what are described as ‘word matrices’ which show how a root word such a ‘sign’ can be altered by various prefixes and suffixes. I have seen Mandy Nayton of DSF demonstrate something very similar, but that was in the context of children in late primary school who had already mastered the earlier stages of a systematic phonics program. Morphology of this kind, coupled with etymology can be particularly helpful for English spelling, given our use of schwas (unstressed vowel sounds that we tend to pronounce as a short ‘uh’, making it hard to determine which vowels to use to represent them). I cannot imagine having a discussion of word matrices with early or struggling readers but then I’m not a reading teacher.

The control condition, motivated reading, is interesting. Children selected a book that was read aloud to them. They then reread it in a group while applying reading comprehension strategies. In addition, one lesson per week focused on vocabulary instruction. It does not appear that any phonics was involved.

Both the motivated reading condition and the SWI condition were delivered by the same set of teaching assistants. The SWI developers decided to script these sessions and they also provided four days of training and fortnightly school visits by the developers. The interventions consisted of three 20 minutes sessions per week for 24 weeks.

The results showed that the effects of motivated reading and SWI were basically the same, both on broad measures such as reading comprehension and on measures you might expect to favour SWI such as a ‘morphological spelling task’. The researchers conclude that SWI is no more effective an intervention than motivated reading.

The researchers suggest possible reasons for the failure of SWI and these include that it was too high level and that the teaching assistants lack of knowledge and confidence in SWI may have reduced its effectiveness (even though fidelity scores for both conditions were not significantly different). This is supported by interviews with the teaching assistants who felt that SWI was more challenging for children to learn.

I don’t find this particularly surprising.

No doubt, these findings will not convince the committed. They may find fault with the design of the study. But it is worth pointing out that these are ideal conditions, designed by developers with an interest in SWI who maintained a commitment throughout the duration of the study. If it doesn’t work as an intervention here then where will it work? What hope is there for rolling SWI out at scale as an initial form of reading instruction?

I would make two more observations. Firstly, motivated reading sounds a lot like whole language to me. Can we conclude, therefore, that SWI is no better than whole language?

Secondly, motivated reading did not appear to contain any phonics. Yet even those who are sceptical of the evidence for phonics will admit that systematic phonics programs are superior to no phonics at all. If so, we may expect a systematic phonics program to beat motivated reading and, from the results of this study, we may therefore expect it to beat SWI.

If you think these observations are a bit of a stretch then fine. The best way forward would probably be to run SWI against systematic phonics in a randomised controlled trial. In fact, I’m unsure why they didn’t do that and why they instead chose to run it against a phonics-free condition we would all expect to be inferior. Go figure.

The sophisticated world of Structured Word Inquiry

Since writing my post about structured word inquiry (SWI), I have encountered a Twitter community of SWI advocates. This has led to two insights. Firstly, I now think I understand why it is so important to this group that systematic phonics should be shown to be no more effective than whole language as a method of early reading instruction. I also think this community exposes a fundamental flaw in the whole SWI project, if that project is about displacing other forms of initial reading instruction with SWI.

Those in the SWI community often point out that they are linguists and what I had not been aware of was the level of contempt in which they hold systematic phonics. To them, phonics is a simplistic mapping of letters and sounds that takes no account of etymology, morphology and so on.

For instance, they may highlight that ‘make’ and ‘making’ have the same initial vowel sound but phonics insists that this is represented as a different grapheme in each of these words, despite the obvious common origin. I have some sympathy for this argument. I guess a systematic phonics advocate would also have some sympathy but would argue that this kind of observation should come later in teaching rather than being something to tackle with five-year-olds. I’m not sure.

As an illustration of the antipathy, one SWI proponent, Gina Cooke, went so far as to describe a list of graphemes from a systematic phonics programme as a ‘lie’, which I understand to be an intentionally false statement:

Cooke also tends to describe advocates of systematic phonics as ‘phombies‘, a portmanteau of ‘phonics’ and ‘zombies’ which strikes me as somewhat pejorative.

I suppose that if you hold a teaching method in such contempt, it is hard to concede any merit in it, even relative to another method you hold in contempt. Perhaps this drives the need to find whole language and systematic phonics equally (in)effective.

Cooke has also taken issue with my writing. I apparently mischaracterised SWI in my previous post. I said that the method focused on teaching morphology from the start. This is apparently a common mistake made by the ignorant and misinformed. Although SWI does teach morphology from the start, it does so in interrelation with lots of other things:

I didn’t think I had claimed that SWI focused only on morphology. In fact, in my original post, I described at some length how Peter Bowers claims SWI also teaches GPCs ‘from the start’. However, I am happy to be corrected by someone who knows more about SWI than I do.

But here’s the thing. There is an awful lot of nuance and fine distinction going on here, in much the same way that many of the SWI community constantly make nuanced and fine distinctions about the nature of the English language itself. For instance, another advocate has the claim that ‘*tion is not a suffix’ listed in her pinned tweet:

I could not possibly comment on the validity of such a claim. I am not a linguist and so I am happy to defer. I would only point out that, in this case, all the dictionaries appear to be wrong.

And here is the rub. For SWI to become a primary means of initial reading instruction, all primary school teachers in the English-speaking world would need to deliver it. Moreover, as far as I can tell, there is no scripted version to follow, or even a highly structured one. And so they would have to know all the stuff that these expert linguists know in order to make all of the correct decisions in real-time while planning and teaching. For instance, they would need the knowledge and confidence to ignore incorrect claims made in dictionaries (if they are indeed incorrect and I am not missing an even more nuanced point that reconciles the linguists and the dictionaries).

My practical experience of working with teachers to deliver improvements in schools – including with the far more simplistic systematic phonics – suggests to me that this seems like a tall order. Primary school teachers will certainly not leave initial teacher education with anything approaching the required knowledge to teach SWI and would perhaps need something like a masters degree in linguistics. Maybe advocates of SWI need to focus on how they can turn it into an instructional strategy that can work at scale. Otherwise, it looks a lot like a game of one-upmanship.