How to fix peer review

Embed from Getty Images

Twitter has been buzzing with discussions about the merits of the process of peer review. This was prompted by a blog post by Dr Pam Jarvis of Leeds Trinity University on the British Educational Research Association (BERA) website. Jarvis draws a distinction between peer-reviewed research and blogging:

“Blogs based purely on the personal experiences and opinions of one person may be presented by some writers (and consumed by some readers) on the same basis as more in-depth, academic peer reviewed publications… Blogging is certainly an excellent tool for both personal and collective reflection. However, all must be fully aware that the meticulous research and peer review processes that are undertaken within academic publishing means that self-published polemic can never be constructed as its equal in a policy advisory role.”

This is an interesting opinion to express in a blog post, not least because there is a slew of academic peer-reviewed research in education that is based largely, if not purely, on the personal experiences and opinions of one person. Such papers arise out of the postmodern shift towards subjectivity and academics have made-up a variety of terms to describe this kind of research. Andrew Old posted an amusing thread of such papers on Twitter and just today a research article came through my RSS feeds in which, “the participants explore the possibilities opened up in a diffractive, post-human space-time-mattering, where memory stories are not taken to be signs of individualized essential selves, but a means of tracing the space-time-mattering of the worlds they live in.” I am no expert in this kind of research, but it seemed to involve turning an anecdote about the classroom into a play. The researchers even made a diorama with the teacher represented by a plastic figure of a police officer.

Perhaps blogs are not the worst thing that can happen. Perhaps peer review is no guarantee that a piece of research has value, particularly in a policy advisory role. Perhaps this is down to the nature of peer review.

Technical review

I have been through a number of peer-review processes and it does not seem to be an optimal system. The fact that established researchers have a role as gatekeepers means that research that challenges established views and theories is always going to have a hard time. I am also unsure why reviewers should remain anonymous. Those who hold a professional opinion about a piece of research should stand behind that professional opinion. If it is a personal opinion then what value does it hold?

Part of the problem is that peer review aims to do a number of quite separate things from basic fact-checking to evaluation of the worth of a piece of research. Some reviewers have passed back to me useful technical corrections and even commentary on grammar. Others have made suggestions about how to present data and aspects of analysis. Comments on the validity of a particular experimental method and what it is capable of demonstrating are also largely technical judgments.

Although I have received a few comments on references, this strikes me as an area of priority. Researchers are evaluated on the number of citations they receive and this can create an incentive for researchers to cite their own papers or those of close colleagues, perhaps with the understanding that they may be cited in return. All papers should therefore have their references reviewed with this in mind. Computer programmes could do some of the leg work and then a researcher could take a view, perhaps deciding whether references are superfluous or redundant and being pretty ruthless in asking for excisions.

In addition, particular journals could publicly lay out their own technical requirements according to editorial decisions. For instance, they could specify whether certain data should be available for double-checking or whether they view it as appropriate to publish p-values. All of these technical aspects could be performed before publication.

Ideological review

However, alongside these technical review functions, peer review also aims to make more subjective evaluations of the value of research and whether it adds anything significant to the field. Reviewers are even free to take issue with the tone of an article. This is where bias has the greatest potential to affect the corpus of work that is published. If you have built your career around a particular model or theory then research that appears to falsify this theory will be unwelcome. Confirmation bias may cause you to dismiss this research by questioning its value or generalisability – which you can do with any research – and you are likely to perceive the authors as impolite, lacking respect for the foundations of your particular field.

Open-access journals

We can disrupt this system a number of ways. Firstly, we need to separate technical and ideological review. It would also seem wise to take technical review out of the direct control of the luminaries in any particular field so that this cannot be easily used as a Trojan horse for ideological review.

In my proposed system, journal publishing would cease to exist as it does at present. Instead, universities and learned societies would set up open-access journals committed to posting as much research as possible online. There would then be a two stage process.

Funds previously spent by university libraries on subscriptions would be diverted into the process of technical review, which would be performed by a mix of technical measures – checking grammar, checking the pattern of citations – and human judgement. The latter could be performed by graduate students as a modestly remunerated role for which they would also gain academic credit. They could even re-run the data, checking statistical models and perhaps learning about these models in the process. A graduate student who wants to check the validity of a non-parametric test that they are unfamiliar with would have a useful prompt for discussion with his or her supervisor.

The reviewers would be public and acknowledged by the journal. Given there are no space constraints involved in online publishing, a journal could commit to publishing every article that meets its publicly available technical standards. This would potentially get around the ‘file-drawer’ problem where studies with null or negative findings tend to remain unpublished and therefore we overestimate the strength of evidence in a particular area. The problem this would then cause would be one of inundation, with journals lacking the capacity to technically review all the papers they are sent. However, waiting lists could be made public and researchers could then decide whether to submit to a journal that is slightly less prestigious but that has a shorter queue. Prestigious journals would have an incentive to expand capacity so that the best research doesn’t go elsewhere.

What about value?

So, am I suggesting no role at all for discussions of the worth of particular studies? Anyone who follows my blog would find that unlikely. Instead, these comments, alongside any further technical comments, should be lodged publicly by leading researchers after publication in a process similar to the comments on a blog. However, this process would be controlled by the journal which would ensure that those making comment were who they said they were and that the paper authors had no veto over who is able to comment.

Given the new publishing landscape, the main metric for researchers would be citations and second-order citations (citations of papers in which they are cited), with publication in a prestigious journal becoming less of a factor than now.


6 thoughts on “How to fix peer review

  1. Having been a peer review a few times, I can tell how I approached this. And, yes, I was anonymous. I took my role as being limited to ensuring that the contemporary literature and the basic texts had been dealt with in the area of study. I also pointed out some infelicitous grammar and expression and sought to draw attention to any logical flaws. What the writer’s hypothese or thesis was, was up to them.

    But you are right, there’s the big temptation to be the intellectual gate-keeper and destroy lines of thinking that undercut one’s own research effort, but that would be termed ‘naughty peer review,’

  2. Chester Draws says:

    Given the new publishing landscape, the main metric for researchers would be citations

    I would prefer — and I know I am a contrarian –is that the main metric would be whether the results were true or not.

    There are plenty of circles where “researchers” cite like minded “research”. Truth doesn’t come into it.

  3. I think you are showing that a peer review process can be captured by inferior reviewers.

    It is not the capturing that is bad but the lack of quality control.

    I think reviewers and authors being anonymous to each other is designed to avoid a risk of quid quo pro. Of course this doesn’t work if the authors are not anonymous to reviewers.

    • Yes. But if everyone was out in the open, authors and reviewers, then quid pro quo arrangements could be analysed by third parties. You could even set up AI systems to search for unusual patterns.

  4. Robert says:

    “Perhaps blogs are not the worst thing that can happen. Perhaps peer review is no guarantee that a piece of research has value”

    I don’t think anyone is claiming that peer review is a ‘guarantee that a piece of research has value’, just as no one is claiming that no peer review (blogs) means no value. Peer review is a sign of quality, not a guarantee, and I think most people in science are quite aware that there’s some pretty low quality peer-reviewed papers out there.

    This is actually the reason why I read blogs. Most blogs I read (including yours) I read because they point me to interesting papers in the scientific literature, *and* provide an easily readable overview of why the paper is worth reading.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.