Evidence for Project Based Learning

Embed from Getty Images

I recently wrote a post about Project Based Learning and how this teaching method was being sold to schools by consultants without the consultants feeling the need to supply any evidence that Project Based Learning is effective.

I was grateful to Bianca Hewes who initiated a constructive dialogue in the comments section and provided links to support PBL. One of these links was to an Edutopia article. With something like Project Based Learning, there is much to research aside from the question of its effectiveness and therefore many of the references don’t address this issue, pointing instead to articles that seek to define PBL or illustrate best practice.

When you narrow it down to questions of overall effectiveness, three references stand out. One links whole-school reforms that involve Project Based Learning to positive outcomes. This is certainly suggestive but it’s hard to isolate the effect of Project Based Learning within a whole school reform that presumably encompasses many measures. The other two papers are from the same issue of the, “Interdisciplinary Journal of Problem-Based Learning,” and review research into efficacy. I wish to take a closer look at these.

Firstly, Problem Based Learning is not necessarily the same thing as Project Based Learning. I guess they become conflated because they apply similar principles and their abbreviated forms (PBL) are the same. The most common enactment of Problem Based Learning is in medical education. I am no expert on this but I understand that small groups of medical students are typically presented with cases to review and analyse, with information supplied to enable them to do this and learn medical skills and knowledge in the process. This seems quite distinct to classes of primary or secondary school students conducting project work, perhaps over a number of lessons and perhaps with an element of choice about which form of the project to pursue.

Medical students will have a higher level of expertise and quite possibly a higher working memory capacity than most school students. Both of these could mitigate the potential drawbacks of problem based learning that are predicted by cognitive load theory. In the Walker and Leary meta-analysis, medical education made up the great majority of the studies with the other contexts also at university level (e.g. teacher education or engineering). The condition that Problem Based Learning was pitted against was usually a pure lecture. We know from teacher effectiveness research that teachers who simply lecture are outperformed by those who interact and ask lots of questions. Nevertheless, the Walker and Leary review paper finds only a small net positive effect of Problem Based Learning (d=0.13 ± 0.025). It is worth remembering that the majority of published education studies tend to show a positive effect and with good reason.

I am wary of comparing effect sizes from meta-analyses but if we are going to venture down this route then there are plenty of interventions that generate larger effects than this and, as John Hattie has pointed out, they tend to involve more active teaching.

There is another point buried in the Walker and Leary analysis that is worth making, even if it appears a little obscure. If a researcher can justify that it is possible for the effect she generates to only point in one direction then she may use something called a “one-tailed” null hypothesis test. If she cannot justify this single direction then she should use “two-tailed” testing. In the case of Problem Based Learning, it is clear that it could potentially generate a negative effect and so two-tailed testing should be used. And yet Walker and Leary state that it was common for the authors of the studies they analysed to use a one-tailed approach. This matters because it makes it easier to find a statistically significant positive result whilst also allowing a researcher to disregard any negative results as statistically insignificant. So negative effects may well be going unreported.

The final paper is by Strobel and van Barneveld. It has the intriguing title of “When is PBL More Effective? A Meta-synthesis of Meta-analyses Comparing PBL to Conventional Classrooms,” and I remember analysing it briefly on my old blog. It eschews the quantitative approach of Walker and Leary in favour of a ‘meta-synthesis’. The authors report effects from a range of studies but only whether they are positive or negative rather than their sizes. This has some merit, given the problems we have interpreting effect sizes but could, of course, be compromised by an under-reporting of negative effects. The abstract states:

“Findings indicate that PBL was superior when it comes to long-term retention, skill development and satisfaction of students and teachers, while traditional approaches were more effective for short-term retention as measured by standardised board exams.”

This is important because, for many reasons, standardised exams are the fairest way of assessing what students have learnt, certainly compared with subjective evaluations. Perhaps all the exams in the studies analysed were simple tests of retention but I have sat few such exams in my life – most ask for some kind of analysis and application. I have written before about the fact that ‘skill development’ in the context of medical Problem Based Learning could mean something like ‘the ability to relate to patients’. Problem Based Learning may ask students to practice this skill a lot by introducing them to patients whereas traditional lectures may not. What can we conclude in this situation? That the lectures are not a good way of teaching something that they don’t attempt to teach?

So I think a few issues remain to be resolved before we use this body of evidence to promote Project Based Learning in schools. Firstly, why should we generalise findings from small-group Problem Based Learning with university students to classroom projects? Secondly, what would happen if we compared projects to interactive explicit instruction rather than lectures? Would we get a positive effect? Finally, are we really okay with the idea that, even when tested by researchers favourable to the approach, Problem Based Learning sometimes seems to be worse for retention of knowledge? Knowledge is quite important, after all.