Academic rigour and The ConversationPosted: May 6, 2017
Here are some claims relating to education that have been made in recent weeks by Australian academics in The Conversation:
- Our declining results in international tests of literacy show us that our 15 year olds can decode but they can’t comprehend. [Source]
- Test results say nothing about teaching quality. [Source]
- A study that looked at how schools develop policies and practices to prevent behaviour problems found the following methods effective… [followed by a list of strategies]
The first two claims are obviously false. The only international reading test for 15 year olds is the Programme for International Student Assessment (PISA). This does not test decoding and so it is simply impossible to use it to infer that ’15 years olds can decode but they can’t comprehend’. Similarly, although there are many problems with using test results to infer teaching quality, it is clearly not true to claim that these results tell us ‘nothing’ about it.
The third claim about behaviour problems seems quite measured in comparison. Indeed, there is a link to the study in question. However, if you take a look at this link then you will see that it is a ‘framework’ that is extremely light on detail. Significantly, a strong causal claim has been made and in order to evaluate such a claim we need to be able to examine exactly what was done, how it was done and what was measured.
The detail is important for a number of reasons. Firstly, some of the points from the suggested list of ‘effective’ strategies look like they originate in progressive education ideology:
When students appear disengaged, staff work in collaboration with students to support the professional learning needs of teachers and develop engaging learning environments…
Staff are committed and have support to enable them to follow up with students the day after if there has been an issue with behaviour. Teachers utilise behaviour management approaches that engage the student rather than punitive approaches that lead to alienation [my emphasis]
The origin of these ideas does not mean that they are necessarily wrong and that these strategies are necessarily ineffective. However, education has an unfortunate history of adopting approaches on ideological rather that empirical grounds. So it is critical to be able to evaluate the evidence supporting such claims.
And this issue is of real, international significance. Many supposedly ‘zero tolerance’ schools would be described by their detractors as using a ‘punitive’ approach and so these claims add to the political debate around such schools. Given that zero tolerance policies have been a feature of some Charter Schools and Free Schools, such claims also add to that debate.
I asked about the evidence in the comments section and received a very interesting response from the author, Anna Sullivan:
“Thanks Greg, that particular link goes to a framework which outlines the findings for this part of the BaSS research. The book that explains this framework is due to be published later this year by Routledge. This is the beauty of The Conversation- research can be disseminated much quicker that the usual peer reviewed publications.”
So the detail of the evidence is in a book that hasn’t yet been published.
The strapline from The Conversation is, “Academic rigour, journalistic flair”. This aspires to a higher standard than most news outlets. If writers are to take the claim to ‘academic rigour’ seriously then I don’t think they should make assertions that are different in substance to the ones that they would be prepared to make in a paper submitted to a peer-reviewed journal.
And no peer-reviewed journal would accept a claim based upon evidence that cannot be examined.
The Conversation is about injecting the voice of academia into the national and international debate. I worry that too many academics take an unintentionally patronising approach to the general public by assuming that the public will not understand the detail and expecting the public to take their assertions on trust. This may have been appropriate for government information films of the 1940s but I think we can do a lot better now.