Who should teachers trust on education research?

Embed from Getty Images

After a brief hagiography from the principal, the education expert strode onto the stage. “Research shows…” he began, before introducing the assembled teachers to his take on this year’s new thing. But all was not well. Somewhere in the audience sat a teacher with a Twitter account. “I don’t think the research actually does show this,” she thought, “but how can I tell?”

Dan Willingham wrote an entire book on the topic of who to trust on education research. It is a difficult issue. The only way to really think critically about research is to read research papers, learn about the methods used and make up your own mind. Some of us are fortunate enough to have the opportunity to do this, but it is impractical for the vast majority of teachers sat in those start-the-year staff meetings.

I am going to offer my own thoughts. Probably the clearest sign that an expert knows what he or she is on about comes from the way they present their arguments. They will tend to take a position on something and they will explain how the research supports that position. However, they will also highlight the limits of the research and the common criticisms of their position, perhaps with their own answer to those criticisms. They will stay focused on the research itself and not try to support their argument by appeals to political sentiment or by attacking the character and motivations of their opponents.

Where will these good experts work? It should be in our universities, but one of the sad ironies of modern education is that education academics often reject the scientific method in preference for qualitative arguments that are essentially based on ideology. At the same time, education academics pour scorn on think-tanks, claiming that you cannot trust the research of think-tanks because it is ideological.

It is best to get off this merry-go-round. Universities produce some good research and a lot of rubbish. Think-tanks don’t have the power of peer-review through the academic journal system to draw upon, but they often shed light on issues that are ignored by universities and that are of more practical relevance to teachers and policy makers. So we shouldn’t get too hung-up on where our experts work, we should pay most attention to what they say.

So back to those points. Why should an expert take a position? Taking a position makes the expert vulnerable because it means that someone could potentially demonstrate that he or she is wrong*. It means that the expert’s own arguments can be analysed and so the expert is making themselves available for scrutiny. By avoiding a position, an expert is suggesting that your ideas and opinions are open for scrutiny, but not theirs. This happens more than you might think in workshop-style sessions or coaching models of professional development that mirror constructivist teaching methods.

It is possible to take a position but see no need to support this with evidence. This can be very effective if it is somehow implied that all teachers who are hardworking, kind, ethical, contemporary or who possess some other desirable virtue must agree with the expert on this particular point. It is effective because by challenging the expert, you become an outcast lacking in that virtue.

I remember a conference I once attended where speaker after speaker derided ‘transmission teaching’. I do not know what they meant by this, but nobody questioned it and I suspect many of those in attendance accepted this as fundamental and obvious, similar to the idea that gravity pulls us downwards.

And anyone who has researched anything in depth will be familiar with the best criticisms of their position. Failing to mention these criticisms is a bad sign. It may signal hubris or it may suggest that these criticisms represent a fundamental threat to the expert. Neither of these is a good sign. If an expert does not volunteer some of the most obvious criticisms then ask what they are. The reaction will be telling.

Finally, it is worth distinguishing between trusting an expert and accepting that they are right. We can all be honestly mistaken. The state of education research is such that there are many holes and nobody can be sure of anything. We are all trying to tidy this mess up into something that makes sense and most of us will have committed errors in the process.

An honest expert knows they are likely to be wrong at some level. A bogus expert will run in the other direction from ever admitting this possibility.

*This is also why it is pointless arguing with someone on Twitter about an issue if the person you are arguing with claims to take no position on that issue. If this happens to you, bail.


23 thoughts on “Who should teachers trust on education research?

  1. In education,…

    Whenever an “expert” says “the research suggests…”, be sceptical but continue to listen.

    Whenever an “expert” says “the research shows…”, you can probably safely tune out.

  2. Not sure I’m with you there about think-tanks. The parallel is with research institutes who receive hefty funding from tobacco companies, and produce research that “shows”, for example, that plain packaging has no effect and that vaping is a healthy way to come off the fags. The whole point of websites like SourceWatch is that donations affect the outcome of the research from think-tanks and similar entities. And to claim (I’m not saying that you said this, but others have) that to criticise research from think-tanks on this basis is somehow “ad hominem” is, well, at my most polite I would say “unconvincing”.

    1. It is ad hominem. If people make bogus claims because they are paid to do so then it should be pretty easy to establish that they are bogus, just as it was, to be fair, with claims about tobacco. I wonder who is funding the CIS to promote systematic synthetic phonics? I wonder which big business interests gain from this? Is it just possible that Jen Buckingham writes what she writes because she believes it to be true?

      1. it should be pretty easy to establish that they are bogus, just as it was, to be fair, with claims about tobacco

        Except history shows that for decades, it was very difficult to undo the terrible damage wrought by Big Tobacco in this area. And why should legitimate researchers have to tie themselves in knots having to peer review “research” that shouldn’t have been put out there in the first place?

        I can’t see inside Jennifer Buckingham’s mind so I’m not going to say whether she believes what she writes to be true. History has also shown that people do write opinion articles, research and other studies that are influenced by their funding, so to say that money doesn’t have an influence is inherently wrong. And I will say is that her enthusiastic championing of school choice and advocating for the supposed “superiority” of private schools (her 2000 study “The ‘Truth’ About Private Schools” is anything but) is on the record – and funding from corporate donors to think-tanks across the world generally translates into “research” that is favourable to school privatisation.

      2. I think there is a difference between dismissing them because of the source and being aware of the source.
        If you treat an individual badly because of the worst a group they belong to has done you are a pretty terrible person if the group is a race.

        Also in this area you have nothing as both academics and privately funded researchers have financial pressures. You could dismiss all academics as most of the research is done by people who can’t afford to go against the prevailing orthodoxy.

      3. To Harry:

        you are a pretty terrible person if the group is a race.

        I’m not sure why you brought race into the argument. You can’t choose your race, but you can choose your political position, habits (smoking, for example) and so on. I’m sure Greg has a handy link to show the weakness of that line of argument …!

        You could dismiss all academics as most of the research is done by people who can’t afford to go against the prevailing orthodoxy.

        If we apply the rules of Popperian falsifiability to that argument, we could say that your statement could be proven to be false if even one academic were shown to go against the “orthodoxy” (which you also haven’t defined) and still keep their job. To whit: Noam Chomsky, Steven Pinker and Jordan Peterson are all academics (Chomsky is retired now) who certainly haven’t “toed the line” in their public statements and yet kept their jobs. I’m sure others can add to this list.

      4. Have you used anything published by the CIS in your PhD? What has your supervisor said about using their material?

    2. Between 1996 and 2010 I wrote dozens of articles that reviewed the research on early reading instruction, including four reports for the Centre for Policy Studies. All of these considered the growing evidence from the cognitive sciences and the results of school effectiveness studies, which strongly supported synthetic phonics. During all but three years of this time, I didn’t even have the benefit of a salary–and save for an article in the Telegraph and another in the Mail, I never got paid a penny for any of this. This didn’t stop some of my critics–most of whom enjoyed comfortable salaries in academia–from bleating on about how I was doing the bidding of my capitalist masters.

  3. One of the education experts that I had to suffer in my workplace (that the school was showering tens of thousands of $, as it happens) showed us a student reflection tool that she had devised. They stand in a line and step forward, backward or stay in their spot to show if they believe the areas that the teacher calls out are a “strength” or a “weakness” of theirs. Something like that, anyway. Supposedly it gives the teacher immediate feedback on their students’ progress. I thought it was utterly ridiculous.

    When the teachers were gathered together again, she asked us what we thought of it (her face showed that she was evidently very proud of it) I put my hand up and commented that it was “based only on self-reflection”. She moved on to someone else but I butted in: “No, I don’t think you understand. It’s purely self-reflection. And that’s why it’s neither helpful nor accurate.” I heard a sharp intake of breath from some of my colleagues. She never called on me again after that, and part of the reason I left the school was because they continued to keep her on as a consultant year in year out.

    1. Interestingly, my current school has very quietly (more or less by universal tacit consent) become a consultant-free zone, and I think it’s because at previous such occasions a critical mass of staff members – mostly the older teachers, I might add – started asking awkward questions when the BS began. Something tells me the senior exec got the message.

      One particularly memorable moment was when a youngish consultant, who didn’t have a clue about, well, anything, got asked a question by an older staff member about a pretty basic Ed concept. The young guy didn’t know what it was, and answered in classic consultantese, “Perhaps, erm, you can tell us more about that.”

      “No, I think that’s your job,” said the older teacher. Red face time.

  4. That tweet by prof Wall is quite amazing. I am trying to imagine an inquiry where it is not possible to reproduce the results leading to a conclusion but you would expect other people to change their practice based on the conclusion.

    I can think of cases with horrible outcomes but that is not really the problem.

    1. education academics often reject the scientific method in preference for qualitative arguments that are essentially based on ideology

      I read the original tweet, and it’s clear that she doesn’t reject the scientific method.

      1. “Scientific method and all it implies are good for some types of education research, but not all. ”

        No not a complete rejection but apparently there is research it can’t be used for according to the prof. Can you help by suggesting what that would like?

        I think she was confusing rcts with the scientific method but she has not elaborated to a query so we are left to guess.

      2. John that was to be my point as well – in fact qualitative research is mainstream in education – it is very difficult to do “gold standard” research – what Greg probably assumes is the ONLY version of scientific method – with kids in schools – with all the complexities and variabilities of socio-economic class, teacher capababilities, students capabilities, students aptitude and attitude, resourcing, leadership, the weather, the time of day – all factors that could affect an outcome.

      3. Again are people confusing rct’s with all quantitative methods? Education is one of the most measured things people do. People will say things like you can’t measure someone’s appreciation of poetry and so on. But that is not the big issue.
        Look at the topics people debate
        Types of reading instructions, the amount of instruction verses discovery, how much homework, use of electronic devices.
        Which are the important topics unreachable with any quantitative approach.

      4. And because it IS so difficult to do gold-standard research (and I’m with Greg on that one – the qualitative stuff is too easy to tweak in favour of one’s own beliefs/prejudices), a certain coterie of Ed academics and particularly consultants ought to be far more circumspect when making the grand claims that they specialise in.

  5. I wonder if there is any research on who exactly carries out education research (or social sciences) and why?

    Is it teachers pursuing their masters/Phd?
    Do they select research after signing up to the course or as a prequisite?

    Who carries out the more complex research? Is it tenured staff with more experience, protection and funds or is it mostly led by low paid researches with only a first degree. Do teachers pursuing their masters/Phd do this?

    What is the ratio of qualitative to quantitative studies?
    What is the ratio of their cost? Does it correlate with above.

    How often are results replicated? Who replicates them?

    Do researchers self-identify in the scientific/critical theory or other camp.
    If they do then are there different answers to the above questions.

    P.s John falsibility is not a counter to an argument to an argument to absurdity (You could dismiss all academics as most of the research is done by people who can’t afford to go against the prevailing orthodoxy.) that simply extended the idea of intellectual provenance (whether appropriate or not). Falsibility would be applied to the claim that some people can not be trusted but others can. If so what are the criteria, the moment you state that criteria it can be investigated for counter examples (falsifiable). Without that criteria it is arbitrary (unfalsifiable). You can try to do that now but you will quickly see how others could challenge it. Money, surely academics need their wage, prestige, isn’t that peer review. That is why your early claims are called an ad hominem by Greg. Some people argue a justified attack is not an ad hominem but this is flawed (and circular). I can legitimately dismiss someones viewpoint because I don’t trust them (we do it all the time) but i should be honest that I have not refuted their point, merely refused to trust it. Justified sin is still a sin. If I talk rudely to someone who is trying to bully me I have still been rude even if I think the sin was necessary. Fallacy’s warn us of likely logical flaws and we should be honest to label our own thinking with them. (I realised I took the liberty of treating a fallacy as a sin when they are obviously different. It was merely a synonym for unpleasant or distasteful act)

    1. Michael, I think the concerns you have raised about the quality of research are more than addressed by the rigorous processes in place at universities. As a post graduate researcher, I can testify to that. Ethics clearance, for one thing, is a massive hurdle to cross and you need to be sure that your research proposal is utterly transparent and that you have declared all interests.

      Think tanks do not have such rigorous processes in place, and it is on the record that donors to certain think tanks want the research to reflect their interests rather than be impartial. Unfortunately, governments can sometimes act in the same way – they don’t release studies they have commissioned because it’s not giving them the answer they want – but those studies do exist, at least, and at some point down the track they can potentially be published.

      I’m glad you acknowledged that the argument put forward above about “all researchers” is absurd. I think bringing in race is even worse.

      Also – and this is for Michael S, above, as well – qualitative researchers are REQUIRED to declare their interests and explain how they arrived at their methodology. If you are using a feminist theoretical framework, for example, you declare this and explain how and why you are using it. That allows the reader to generate their own context. Also, you need to back up your theory with a supporting literature review – to show you’re not just pulling it out of your butt.

  6. I don’t believe that universities are a hot bed of unethical research I was pointing out that dismissing arguments from other locations is a ad hominem. We would be fools to not consider the provenience of arguments but we must engage with the intellectual merits of each and every argument to make progress. If I distrust an idea because it came from a think tank/university/guy in the pub I should be honest and Ideally this should be at the end. That way I can state (I agree/disagree with this idea because….. however I don’t understand/can’t counter this argument however I distrust the source and need to look for confirmation/counter arguments from other sources). Finally Universities (specific ones I suspect) are as vulnerable to self-affirmation, institutionalization and confirmation bias as other organisations and this can not be assumed to be picked up in an ethics committee anymore then a boardroom.

    If literature reviews have taken place then elucidate those findings and explain the argument. The key issue here is around what is the scientific method and what are the alternatives that are being used. This would be illuminating. I know critical theorists reject a focus on empirical and impartial data to emphasis context in direct opposition to a classical ideal of science. I understand the importance of qualitative data but that information is easily gained but of little use in answering precise comparative questions. What works better in this particular situation with this particular objective. In short my understanding is that there are competing ideologies with differing goals and I very much prefer one over the other depending on my objectives. I have read of many people complain of scientisim or a simplistic view of science (including several scientists I admire) I have yet to here what the alternative formulation is (which may change my mind)

    Personally my objectives are I want to know were to look for the best approaches to raising my students abilities, outcomes and knowledge, so i need high quality quantitative studies that are often but not always gold standard RCT’s. (The DISS study is an example of phenomenal quantitative research which is not a RCT though). These are rare due to their expense and rarely read or communicated.

    Hope this helps.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.