Gatekeeping education research

Embed from Getty Images

An amusing post has been published over at the Australian Association for Research in Education’s (AARE) blog site. It’s amusing for the song beneath the words; for the kinds of values that researchers will come out of their corners and fight for.

This time, Emma Rowe and Trevor Gale pour cold water on what might initially seem like an odd target: a promise made by the Australian Labor Party to set up an educational research unit if it wins the next election. You might think that education researchers would welcome such a move. But not in this case.

There are some valid concerns about Labor’s new policy. For a start, it naively seeks to ‘take the politics out of the classroom’. This could be difficult in an environment where armies of researchers are busying themselves trying to insert as much politics as possible into the classroom. Who will run this institute? Will it be captured by postmodernists? Will it go on unicorn hunts like Britain’s Education Endowment Foundation (EEF) and its promotion of Philosophy for Children? Why do we need a new research institute when we already have one in Evidence for Learning (E4L)?

But these are not the concerns of Rowe and Gale and the way they seek to dismiss the policy is instructive. Apparently, education research is thriving in Australia already and we know this from international rankings of universities. Right.

Australian education research is so good, it seems, that ideas about ‘rich tasks’ that have been developed in Australia have been adopted in Scotland as part of its disastrous ‘Curriculum for Excellence’. Which is hardly a recommendation.

And that is not all. As I highlighted last month, Labor’s Tanya Plibersek called for the new institute to utilise randomised controlled trials (RCTs) of the sort conducted by the EEF and E4L. This call sheds some light on the following statement about research institutes by Rowe and Gale:

“They can also impose one particular way of doing research as the ‘gold standard’. That is what’s happening in the UK with the government-sponsored Education Endowment Fund, with its exclusive bent for random control trials, despite these being discredited in the social world of education.”

That last section, ‘despite these being discredited in the social world of education’, contains a hyperlink. The convention in this kind of post is that the hyperlink takes you to the evidence supporting the claim being made, but in this case it simply takes you to an audio clip of Gale speaking, where he basically says that there are multiple ways of knowing and talks about a ‘playful’ book chapter he has written that expands on this.

So it’s a bit like me saying that ‘cats have been discredited as pets’ and then attempting to support this claim by providing an audio clip of me explaining why I think cats are rubbish pets.

It might seem like a small issue to highlight, but it is related to a much wider point. When you reject the scientific method in favour of something else then these alternatives are not equally robust. Other ways of knowing cannot lay claim to the kind of authority that well-designed experiments accrue and often simply look like opinions heaped upon opinions, frequently with reference to the opinions of a French philosopher somewhere along the way.

Anyone who cares about education, cares about things we can do that will make it either better or worse. These claims are testable in principle, even if such tests may be hard to conduct. The fact that education involves humans does not invalidate the experimental method any more than it invalidates it in medicine. Neither does the fact that we can only make predictions about average effects. Scientists have been comfortable dealing with uncertainty for centuries (in the audio clip above, Gale makes the incorrect claim that medical research is ‘certain’ when in fact it is probabilistic).

Of course, experiments also have the nasty habit of giving the wrong results. If you believe in something that is unlikely to be effective then the last thing you want is some bean-counter running randomised controlled trials and proving you wrong. And this, I think, is what it is ultimately about.

Yes, Australian education academics have been pretty good at accruing resources. They have been less effective at communicating their message. And now, some start to feel their feet twitch as they sense the rug being pulled from under them. This is why they need to ‘just ask questions’ about grassroots movements like researchED. It is why they need to lodge complaints about education bloggers. And it is why they don’t want a new source of authority to be created that runs experiments.

As Rowe and Gale conclude:

“If the Labor Party or the Australian Government are seriously looking for ways to move closer towards research-informed teaching and schools, they should start by promoting and distributing the world class educational research that Australian educational researchers are already producing.”

Well, they would say that, wouldn’t they?


End note: As an aside, Rowe and Gale make an odd point about funding that I struggled to connect to their main argument. While making this point, they reference New Zealand as being one of the highest performing countries in PISA literacy. As far as I can see, New Zealand has been declining on this measure since about 2009, which neatly highlights that it’s not raw rankings we should pay attention to in PISA but the direction of travel of each education system.

Advertisements

13 thoughts on “Gatekeeping education research

  1. An interesting post and a fascinating topic. I will preface my comment by saying that I am not, have never been and never will be an expert on the design and conduct of research.

    In the Jul-Sep edition of the Journal of Family Medicine and Primary care, Leung suggest that there is as much discussion around these issues in medicine as there is in education.

    “In general practice, qualitative research contributes as significantly as quantitative research, in particular regarding psycho-social aspects of patient-care, health services provision, policy setting, and health administrations. In contrast to quantitative research, qualitative research as a whole has been constantly critiqued, if not disparaged, by the lack of consensus for assessing its quality and robustness. This article illustrates with five published studies how qualitative research can impact and reshape the discipline of primary care, spiraling out from clinic-based health screening to community-based disease monitoring, evaluation of out-of-hours triage services to provincial psychiatric care pathways model and finally, national legislation of core measures for children’s healthcare insurance. Fundamental concepts of validity, reliability, and generalizability as applicable to qualitative research are then addressed with an update on the current views and controversies.”

    Surely one can gather any data one wishes, but the claims made about knowledge arising will vary in veracity.

    When you say this…

    “These claims are testable in principle, even if such tests may be hard to conduct. The fact that education involves humans does not invalidate the experimental method any more than it invalidates it in medicine.”

    I think you do both yourself and Gale a disservice when you refer to “certain” and “probabilistic”. There clearly are aspects of medicine which are clearly related to chemical and/or biological sciences where research results can be generalized with a high degree of validity/reliability. The results of testing drugs on cells for instance can produce results with a high degree of certainty and which for all intents and purposes could be described as “certain” when compared with the results arising from qualitative research.

    There appears to be a great deal of qualitative research, the results of which inform clinical practice in medicine especially in a psycho-social context.

    I believe the following provides considerable insight into your article….

    “Of course, experiments also have the nasty habit of giving the wrong results. If you believe in something that is unlikely to be effective then the last thing you want is some bean-counter running randomised controlled trials and proving you wrong. And this, I think, is what it is ultimately about.”:

    You seem to have a preconceived idea that RCTs and statistical inferences can be applied almost universally, if not universally. You like to draw parallels with medicine, but as is often the case in the bloggershpere you seem only to focus upon those aspects of medicine in which “your preferred methods” produce insights into populations with generalisable results.

    I have read around the issue and I see a good deal of debate among medical professionals and academics as to whether medicine has become too reliant upon quantitative results where the use of such results is likely not valid. This debate in medicine for me tends to mirror the debate in education. Of course there is room in education for RCTs and research based upon the chemistry and biology of human beings, there is room for effect sizes and treating people as populations even while dealing with them as individuals.

    Much of this discussion appears to me to revolve around whether we wish to treat learners as individuals and try to develop solutions which help them or whether we wish to find best general solutions for populations and impose these on individuals for whom they may not apply.

    Surely there is a place for both.

    This article for me comes across as bickering, as oneupmanship, as male chest beating…..”my methods are better than yours”.

    It seems clear to me that RCTs can be used effectively in some situations and qualitative research in others. I think science and maths teachers should consider the debates within medical research / practice before cherry picking those areas in which the similarities in context facilitate bickering and pomposity.

    Just my views as a non expert interested observer.

    1. Imagine for a moment that medicine had not embraced the the quantitative methods of science but instead followed philosophical and ideological approaches to determining what treatments were used. This is the state of education today. For over half a century the results of cognitive science/psychology that apply to teaching and learning have been largely ignored along with the results of large-scale in school quantitative investigations of teaching methods. Unquestioned ‘science-like’ but later disproved ideas like brain gym and learning styles to name a few, have swept through schools, one after the other. So an acknowledgment of scientific evidence and what it has to offer is long over due in education, unlike medicine, where, if what you say is correct, there is a movement toward acknowledging the value of qualitative approaches.

      1. not embraced the the quantitative methods of science but instead followed philosophical and ideological approaches

        I’m puzzled – are you suggesting that the opposite of “quantitative” is “philosophical and ideological”?

      2. For over half a century the results of cognitive science/psychology that apply to teaching and learning have been largely ignored along with the results of large-scale in school quantitative investigations of teaching methods.

        Examples?

    2. Wasn’t very clear was I? What I meant to say was, had medicine not embraced an evidence-based scientific approach but instead continued to follow methods based on untested beliefs. Examples for education ignoring science – how about the belief in transferable skills such as critical thinking which experiments in cognitive science were showing as early as the 1940’s were dubious. We have the “testing effect” as an aid to improving recall – around for a very long time, hardly acknowledged in classrooms. Then we have learning styles which is simply another form of “cognitive styles” which have been investigated since the 50’s and no evidence found in support to this day. Then the debate between phonics and whole word approaches in teaching early reading when as early as the 50’s studies were showing phonics was important – at the present day there is incontestable evidence around systematic synthetic phonics being a critical ingredient in reading instruction yet it is still ignored in many Western reading programs. Then we can look at the evidence about teaching underprivileged children accumulated in the massive longitudinal study Project Follow Through, the results of which were swept under the carpet because they were not ideologically acceptable to the educational establishment.

  2. Also on New Zealand, didn’t Pirls show that they had a large rate of below expected level readers that was only growing?

    Funny example when you’ve just written an article trying to say there is so much good evidence around and your concerns that people might try and ‘bend’ some evidence. Just an observation though.

    1. Just for the record – from NZ’s Education Review Office documents: “According to recent Programme for International Student Assessment (PISA) data, the reading achievement of our 15 year olds is on a
      steady decline and New Zealand is one of very few countries in which the
      mathematics and science achievement of 15 year olds is on a trajectory of
      accelerated decline. PISA data also shows that within the same school, young
      people can experience widely divergent opportunities to learn. This within-school
      inequality is amongst the highest across the countries that participate in PISA
      assessments.

  3. JP-Example: The Collier study of program models that serve English learners in the US: A pullout program leaves the learners in worse shape than providing no program. Yet most programs are still pullout.

  4. “. ..New Zealand is one of very few countries in which the mathematics and science achievement of 15 year olds is on a trajectory of accelerated decline. ”

    And we’re fixing it with “Relationships”

    1. Indeed Goddinho.

      And it started with the dreadful Numeracy Project, which had the solid backing of educational academics and researchers. They got exactly what they wanted, and it’s results were baleful.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.