Conducting sound education research should be as easy as ABC

I’ve had an idea.

In education, we are surrounded by pseudoscience. There are spurious diagrams of brains or eccentric research approaches. And all of it gets wrapped up in politics. Nobody would claim that it’s somehow ‘right-wing’ to dismiss homeopathy – indeed, alternative medicine is often associated with the privileged classes. Yet if you challenge alternative education then you can expect to attract this label.

Medicine is not perfect, but the reason it has made more progress than education is that it has a sounder evidence base. So that’s what we need. Unfortunately, this is where we hit a major problem: Everything works.

It was John Hattie who made this claim in his 2009 book, Visible Learning. All education interventions appear to work due to the inherent problems with designing studies. It is very hard to design an educational experiment where the participants are blind to the fact that they are receiving the intervention. So this will affect expectations. The teacher or students might try a little harder or simply think about subject content a little more.

It was also Hattie who proposed a way around this. If everything works then let’s look at the size of the effect. By comparing effect sizes, we can see what works best. These will be the interventions where the effect size is large enough that it is unlikely to have arisen due to the subjects’ expectations. Hattie set an arbitrary cut-off for effect sizes of 0.4 of a standard deviation.

The trouble is that you can’t really do this. Effect sizes from different experiments aren’t really comparable in this way. For instance, the effect size will be larger with small children or with a selective cohort of students. 

So here’s the idea. Let’s mobilise the resources of groups like the Education Endowment Foundation in the U.K. and Evidence for Learning in Australia to run a different kind of trial; a trial that follows the model of one of my favourite papers.

Instead of having a control group and and intervention group – an AB design – we should run trials with one control group and two competing intervention groups – an ABC design. Both interventions would need to be supported by researchers who are committed to them and both would need equal resources. We could then see which of the two interventions works best. Comparison would be fair because it would be within one experiment.

Good candidates might include running Reading Recovery against a systematic synthetics phonics programme or running ‘productive pedagogies’ against a programme rooted in teacher effectiveness research.

None of this would completely fix the problem of pseudoscience. You’d still see eccentric articles in The Guardian and the proponents of alternative education would rant and rave about ‘positivism’ and politics. But we would start to build an evidence base that could be drawn upon by reasonable teachers and policy makers who haven’t yet hitched themselves to the wagon of woo. Slowly and quietly, we could edge towards a more evidence-based profession.

Advertisements

6 Comments on “Conducting sound education research should be as easy as ABC”

  1. Terry Byers says:

    Greg
    What you have described is a multiple intervention approach use in a single subject research design (SSRD). SSRD is often applied in the applied health sciences to measure the impact of a treatment (or intervention) on an individual or group of patients. The low validity A (baseline)/B(intervention) are often used because the context (or time) just allow the more rigorous ABA (withdrawal), multiple intervention (ABC) or multiple baseline. Originally the field focused on the within-group comparision of the same unit of analysis (individual or group) over time. Now, it is expected to have both within- and between-group comparison to a control group ( not involved in the intervention) to better account for a myraid of confounding variables (but the design cannot account for all as in a RCT). Finally, there are various analysis techniques ranging from simple visual analysis to OLS regression and parametric calculations (i.e. TauU is one such).

    Such an approach in a school setting is possible (i used it in my PhD), but the nature of confounding variable control makes longitudinal time-series studies beyond the school year difficult (ie student in class A with teacher A in 2016, but in class B with teacher C in 2017, let alone diff curriculum, assessment items and maturation-why you now need a control group in SSRD).

    Terry

  2. Terry Byers says:

    Greg
    What you have described is a multiple intervention approach used in a single subject research design (SSRD). SSRD is often applied in the applied health sciences to measure the impact of a treatment (or intervention) on an individual or group of patients. The low validity A (baseline)/B(intervention) are often used because the context (or time) does not allow the more rigorous ABA (withdrawal), multiple intervention (ABC) or multiple baseline designs. Originally the field focused on the within-group comparision of the same unit of analysis (individual or group) over time. Now, it is expected to have both within- and between-group comparison to a control group ( not involved in the intervention) to better account for a myraid of confounding variables (but the design cannot account for all as in a RCT). Finally, there are various analysis techniques ranging from simple visual analysis to OLS regression and parametric calculations (i.e. TauU is one such).

    Such an approach in a school setting is possible (i used it in my PhD), but the nature of confounding variable control makes longitudinal time-series studies beyond the school year difficult (ie student in class A with teacher A in 2016, but in class B with teacher C in 2017, let alone diff curriculum, assessment items and maturation-why you now need a control group in SSRD).

    Terry

  3. ALEX O'MAHONY says:

    Wonder if you an help me out here? I am a primary school teacher, in a working class area of Dublin, who became convinced of the value of SSP at the turn of the century. In 2001 I entered teaching from the private sector at the age of forty, I looked at our kids, I saw their weaknesses and through the work of the Reading Reform Foundation I discovered phonics. I began to experiment with this approach to the teaching of reading and I got promising results.

    In Ireland the education debate is not as fierce as it is in England and generally speaking we are allowed to get on with our jobs. In my school, our principal has supported the teaching of phonics because we have produced clear evidence that it works. Our scores in literacy and for that matter maths, have improved markedly over the years. The evidence we have in support of SSP is strong and it would be rediculous to risk the gains we have made by moving away from SSP.

    With this in mind I wonder if you would be able to do some digging into what has happened in Clackmannanshire, Scotland? The famous Clackmannanshire research into SSP had a massive impact on the Rose Report and the introduction of the Phonics Screening Test. From my own personal viewpoint, it convinced me that the changes I was pushing for in my school were correct. Yet, I am now hearing that post the Clackmannanshire research, SSP in Scotland was never really supported by the Scottish government. In addition, I am led to believe a mixed methods approach has since been introduced into Clackmannanshire itself!

    If this is true I would ask the question as to what hope is there for education research triumphing over ideology?

  4. […] is a variant of an ABC design that pits one intervention against another and I believe that this is the best way forward for large-scale RCTs such as those conducted by the Education Endowment Foundation in the U.K. and […]

  5. […] Education Endowment Foundation (EEF) seem keen to fund such studies and it is a major reason why I have argued for more ABC designs where two competing interventions are compared against each other and a […]

  6. […] on track. We can’t really eliminate expectation effects in studies like this, but the use of ABC designs, where one intervention is pitted against another, would be a step in the right […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s