A guide to reading research papers

It can be hard to make sense of an education research paper and there are certainly those who think teachers shouldn’t worry their pretty little heads about them. Yet if we intend to claim our profession back from quixotic consultants then it’s something that more of us need to do. 

There are plenty of dense papers written that aim to show that the curriculum is neoliberal or that tests are phallic. There are also many studies that seek to find correlations. I’m not going to discuss these here. Instead, I am going to look at experimental research.

Experiments offer us the chance of us making meaningful inferences – although it’s only a chance. Experiments need to compare a particular approach with an alternative. For instance, one group of students might be taught maths in the morning and the other in the afternoon. You can then judge the difference that this does or does not make.

1. Read the abstract, methods then discussion

Once I have established that a paper has been peer-reviewed (ERIC can help determine this – I really wouldn’t bother with papers that haven’t passed this level of professional quality control), I read it in this order: abstract, methods, discussion. The results section often becomes pretty technical and hard to follow but it’s still worth a look in case something pops out – sometimes there are interesting non-effects hiding in there.

What claim is being made in the abstract? Key results like effect sizes and tests of significance should be in there. If not, we might wonder why. We’ll return to these later. 

Is the claim that students simply enjoy a particular teaching method? If so, I wouldn’t bother too much. Anything can be made enjoyable or boring for the duration of most experiments. Focus more on claims that one method leads to more or better learning. 

2. How are participants selected?

The best way to select participants for a study is to do so randomly. It is then a matter of chance whether students end up in the experimental or comparison groups. We can still make useful inferences from non-random studies but we do have to be aware of possible bias. 

If Mrs Smith’s class get the experimental biology teaching condition and Mr Jones’s class do not then in what other ways do these classes vary? 

We might try to control for this by doing a pre-test and showing that both groups start out with the same science knowledge. But what if it turns out that Mr Jones’s class is scheduled at the same time as Latin so all the Latin students are in Mrs Smith’s class? And the biology topic involves lots of Latin words…

It’s almost impossible to anticipate all of the ways in which groups may systematically vary which is why random selection is better.

3. What does the comparison condition look like?

Possibly the biggest problem that we have is poor quality comparisons. This links to point 4 on ensuring a fair test.

Imagine researchers go in to a school and ask for teachers to volunteer to take part in a study. These teachers end up delivering the experimental condition and are compared against the other teachers who deliver the old course. Teachers who volunteer may well be more enthusiastic or interested in the process of teaching. They may simply be better teachers. This, rather than any detail of the experimental approach, might account for a positive result. 

Many studies tend to use comparison methods that do not represent the best possible alternative. For instance, ‘active learning’ conditions involving college students are often compared to lectures and find a positive effect. This is fine as far as it goes but we are not justified in inferring that active learning is more effective than direct instruction because direct instruction is not lecturing

4. Is only one thing being varied between the conditions?

A key principle of science is that we vary only one thing at a time. In the social sciences, this prompts the question: what is a thing? Often a whole package of measures is compared to not doing anything at all. This may still be of use but it can also lead us into error. 

For instance, imagine our package involves giving teachers professional development (PD) on maths and then asking them to teach using an abacus. If we see an effect then it’s tempting to attribute this to the use of the abacus. Yet we know that simply giving teachers PD – making them think more deeply about maths teaching – is likely to have an effect. We might reason that this will be even more likely in elementary schools where teachers are often non-specialists. A fair test of the abacus would need to have a comparison group where teachers received the same amount of maths PD but didn’t use the abacus. 

You see a similar problem when children in an intervention group simply get more of something e.g. additional reading tuition. Is it the form of the tuition or just the amount? I’ve also seen studies where the experimental and control groups are taught different topics in different ways. It’s hard to conclude anything from that.

5. What’s the outcome test?

My first principle of educational psychology is: students tend to learn the things you teach them and don’t tend to the learn the things you don’t teach them.

Imagine you ran a study where one group of students were given experiments to conduct involving balls and ramps in order to learn physics concepts and the other group was taught these concepts through teacher exposition. The students are then set a test that is all about conducting experiments with balls and ramps. The first group perform better and so the researchers conclude that the first approach is better for learning scientific inquiry skills. 

This is flawed because the comparison group have had no opportunity to learn the thing being tested. 

Often, research studies will include examples of test questions in a description of the method or appendix so look out for these.

6. Interpreting results

Unless you have pretty good statistics knowledge, you are probably limited to looking for statistical significance and effect sizes. 

Statistical significance is usually given by a p-value (or confidence intervals which are equivalent). If p<0.05 this means that if the experimental condition actually had no effect then the chance of obtaining the results that this experiment obtained is less than 5%. This is useful. However, if the researchers ran the experiment 20 times then you would expect one result like this (which is why publication bias is such a problem). Similarly, if 20 outcomes were measured, you’d expect one to have p<0.05 by chance. Very large numbers of participants are also likely to generate a small p. Finally, if the chance of the experiment having no effect is very small then the p-value is pretty meaningless.

Effect sizes have been much debated. I don’t think we can treat them all the same in the way that John Hattie does – effect sizes from poorly designed experiments tell us little because the effect is probably a result of the poor design. We also need to be aware that effect sizes are smaller with older students than younger ones and with standardised tests as opposed to tests created by the researchers. There is nothing shady about this last point – standardised tests often don’t focus as much on the topic studied as part of the experiment. We just need to bear these points in mind.

Potentially, we can use effect sizes to make up for the problem of poorly controlled studies. If reading intervention A generates a larger effect compared to doing nothing than reading intervention B then we might conclude that A is superior to B. This isn’t as strong as if we had run both interventions in the same experiment.

You can read more of my thoughts on statistical tests here

7. A meta-test

Finally, if you can’t establish whether a paper has been peer-reviewed or you can’t answer my points 2-5 above then I would take this as a bad sign.

The practice field

Why not try my analysis on the papers below:

Paper 1

Paper 2

Advertisements

7 Comments on “A guide to reading research papers”

  1. I have just finished an MSc in using technology in sciences lessons so this was an interesting read. You express many things I have discovered over the last 6 months. I have one extra tip for those short on time. Once you have read the title of a paper move straight to the last quarter of the abstract. Well written abstracts should contain well defined conclusions in these last few sentences. On collecting reliable data; as science teachers we always encourage students to identify all the variables. Then you change one variable, measure another and keep the rest the same. It is amazing how many papers I read that didn’t stick to this simple rule teach in school. I fully appreciate it is often difficult to isolate one variable particularly when no single teacher or set of students are the same. However, researchers sometimes don’t even try to isolate a single variable and it makes it difficult to conclude anything. What you are left with is some fluffy comment like, students were engaged and the teacher gained experience in using intervention X. One study I read included using an app on a tablet, blogging and teacher instruction. I am not sure what can be concluded from tests such as these??

    Thanks for comments on effect sizes. Maybe they are not perfect but there doesn’t seem to be anything better to compare results of different studies. I guess you have written about randomized controlled trials elsewhere? Finally, I would suggest to any budding academics seek out literature reviews on their chosen subjects. If you can find them, they are a great place to start your reading.

  2. Reblogged this on toptenteacher and commented:
    Many helpful tips on checking quality of papers which express many of my findings from the last 6 months of completing my MSc.

  3. Reading the introduction section is also useful because it tells you what conceptual framework the authors are using.

  4. […] So . . . this is why I would suggest that the “studies” Boaler, a contentious enough figure in her own right, cites in her opinion piece are questionable. For a concise approach to evaluating educational research, Greg Ashman outlines a useful approach here. […]

  5. […] was deze blog hier in het Engels gepubliceerd door @gregashman. Met zijn toestemming heb ik die in het Nederlands […]

  6. […] was deze blog hier in het Engels gepubliceerd door @greg_ashman. Met zijn toestemming heb ik die in het […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s