Open Peer Review

Image: Pexels (CC0)

What is Open Peer Review, and how is it different from other review models?

Why is Open Peer Review Important?

Open Peer Review has become a popular topic of discussion and debate. Though not as prevalent as blinded models, and often viewed as a progressive or alternative approach, Open Peer Review isn’t really new. In fact, Open Peer Review in one form or another has been part of the conversation in scholarly communications since the 1980s (McGiffert, 1988). Today, many scholarly journals employ versions of Open Peer Review in their day-to-day practice, including BMJ, BMC, Royal Society Open ScienceNature Communications, and, of course, the PLOS journals, to name just a few. The odds are good that you’ll be invited to participate in an Open Peer Review process–if you haven’t already.

What is “Open” Peer Review?

First, a disclaimer: there is no universally accepted definition of Open Peer Review. A 2017 systematic review identified seven elements of Open Peer Review currently in use in 22 different combinations and configurations across scientific publishing (Ross-Hellauer). Broadly speaking, Open Peer Review refers to some combination of openness in: 

  • author or reviewer identities
  • publishing peer review content
  • participation of a wider community in peer review
  • interaction between authors, editors and reviewers
  • open review before publication through preprints
  • post-publication commenting
  • platforms that de-couple the peer review process from the publication process 

Some of these examples of peer review can be seen in the infographic below. Here we’ll be talking mainly about published and signed peer review, where review content is published, and reviewers sign their reviews.

Types of Peer Review

In order to understand signed and published, it helps to see how the model fits into the larger peer review landscape. 

Peer review is a pillar of scientific communication, the mechanism we rely on to ensure that published research is thoroughly vetted and scientifically valid. For that reason, we tend to think of peer review as a monolith–iconic, stable, and consistent. In fact, journals have used many different forms and applications of peer review over the past twenty years, often in parallel.

 Download

Benefits of signed and published peer review

Proponents of signed and published review argue that increasing transparency in the peer review process leads to a better understanding of published research, more constructive peer reviews, and well-deserved credit for reviewers.

✔ Enriching the scientific record

Signed and published peer review helps contextualize research and gives readers the benefit of additional expert opinions. Letting readers see the questions reviewers raised and how the authors mitigated them gives insight into the limits of the study. Publishing peer reviews also reinforces the validity of the article by exposing the rigorous vetting process it has undergone prior to publication.

✔ Credit for peer reviewers

Peer review is challenging, time consuming, and too-often unacknowledged. Signed and published peer review offers reviewers an opportunity to claim credit for their work. It’s also a first step toward elevating peer reviews to scientific outputs in their own right.

✔ Educational tools

Making peer review publicly available creates a database of examples for students to reference as they begin to participate in peer review.

✔ Accountability

When peer review is transparent, potential competing interests are more readily apparent and everyone participating in the peer review process (reviewers, editorial board members, and authors alike) is publicly accountable for their actions.

✔ Quality of feedback

Research suggests that signed and published peer reviews are at least as good as, and may be slightly better than, blinded models. Two BMJ studies on signed review (Van Rooyen, 2001) and impact of possible public posting (van Rooyen, 2010) found no difference in quality; however, other research found improvements in specific areas like constructive feedback, comments on methods, length of review, and substantiating evidence to support the comments (Kowalczuk, 2013; Walsh, 2000; Bornmann, 2012; Mehmani, 2016).

Concerns about signed and published peer review

Anonymity in peer review is designed to protect against bias and retaliation, though experience has shown that this isn’t always effective. Detractors of signed and published review express concern that it could compromise the integrity of the peer review process or lead to bullying and recrimination. However, research suggests that this isn’t always the case, and that potential risks can be mitigated.

✘ Invitations

Reviewers may be less willing to review if they know their review could be published, either with or without their name.

In 1999, BMJ reported that mandating signed peer review made peer reviewers 12% more likely to decline invitations (van Rooyen, 1999). However, this reluctance seems to have decreased over time. More recently, The EMBO Journal reported no change in reviewer acceptance rates (Pulverer, 2010) after implementing signed and published peer review, while at The European Journal of Neuroscience, only 18 of 3293 invited reviewers declined due to signed peer review (Bolam, 2017).

✘ Retaliation

There could be negative career consequences for critical reviewers, especially for junior researchers who are dependant on senior scientists for opportunities and advancement.

Harassment and bullying do occur in science–and blinded review is not necessarily protection. According to some studies, blinding can fail in as many as 10% (van Rooyen, 1998) to 32% (Justice, 1998) of cases. Journals and funders can have a positive impact by explicitly naming retribution against peer reviewers as misconduct with consequences (Bastian, 2018), and by making the whole process more transparent and open to public scrutiny, thereby reducing opportunities to engage in negative behaviors. Ultimately though, it’s up to each individual reviewer to decide whether they’re comfortable signing their name to a review, or participating in a published peer review process.   

✘ Rigor

The stringency of criticism could diminish, either because reviewers are less comfortable giving critical feedback, or because the reviewers who would have had criticisms simply decline to participate in peer review. Again, as discussed above, the research shows no difference in reviewer recommendations to accept or reject between blinded and signed reviews (van Rooyen, 1999)–but more investigation is needed.

Who is right?

There is no one right way to do peer review. 

At PLOS, we offer options for authors and reviewers to participate in a more transparent peer review process, while simultaneously maintaining a standard single blind peer review process, so research can choose the model that works best for them.

There is a real need for more investigation into peer review. It’s our hope that by making signed and published peer review optional across our portfolio and sharing the outcomes with the broader scholarly community, we can provide a broader dataset for analysis.

How do I complete an “Open,” signed, or published peer review?

First, know where you stand. Take some time to review the research, talk to colleagues you respect, and consider whether you want to sign your reviews as a general principle. Remember, just because you have a personal philosophy doesn’t mean there won’t be occasional exceptions–but at least this way you’ll feel confident about your choices and the reasons behind them.

The practical aspects of completing a peer review remain the same, regardless of the review model a particular journal might happen to employ. There are three key stages to completing a peer review, Open or otherwise.

  1. Responding to invitations. Accept an invitation to review only if you have the necessary time and expertise, and can provide an objective review. Journals will let you know what peer review model they practice before you agree to review, usually in the invitation letter itself, or on their website.
  2. Reading the manuscript. Before you read the manuscript, make sure that you understand the journal’s criteria, so you know what to watch for as you make your assessment. Read the manuscript through once for a general understanding, and a second time taking notes as you go. Pay special attention to the research question, methods, and conclusions, and review the figures and tables in conjunction with the results.
  3. Writing the Review. Begin with a summary of the research and your overall impression, before moving on to discuss specific areas for improvement and any other points. There is no need to do the authors’ work for them by suggesting line edits, specific manuscripts to cite, or experiments to do. Just let the editors know where you had questions or concerns.

How does signed and published peer review work at PLOS?

We’ve adopted a modular, opt-in approach to transparent peer review. Reviewers choose whether to sign their names to their reviews. If accepted for publication, authors can choose whether to publish the peer review history alongside the final article.

Read more about peer review history at PLOS.

Citations:

Bastian, H. (2018, March 22). Signing Critical Peer Reviews & the Fear of Retaliation: What Should we do? [Blog post]. Absolutely Maybe.

Bolam, P. (2017, September 14). Transparent Review at the European Journal of Neuroscience: Experiences One Year On, [Blog post].

Bornmann, L., Wolf, M. & Daniel, H.D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91(3): 843-856.

Justice A.C., Cho M.K., Winker M.A., Berlin J.A., Rennie D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial, JAMA. 280(3):240-2.

Kowalczuk, M.K , Dudbridge, F. , Nanda, S., Harriman, S.L., & Moylan, E.C. (2013). A comparison of the quality of reviewer reports from author-suggested reviewers and editor-suggested reviewers in journals operating on open or closed peer review models. F1000Research.

McGiffert M. (1988). Is Justice Blind? An Inquiry into Peer Review. Scholarly Publishing, 20(1): 43-48.

Mehmani, B. (2016, September 22). Is Open Peer Review the Way Forward? [Blog].

Pulverer, B. (2010). A transparent black box. The EMBO Journal, 29(23):3891-3892.

Ross-Hellauer, T. (2017). What is open peer review? A systematic review [version 2; peer review: 4 approved]. F1000Research, 6:588 

van Rooyen, S., Godlee, F., Evans, S., Smith, R., Black, N. (1998) Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA, 280(3):234-7.

van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999, Jan 2). Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ, 318(7175):23-7.

van Rooyen, S., Delamothe, T. & Evans, S.J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ, 341:5729. 

Walsh, E. Rooney, M. & Wilinson, G. (2000). Open peer review: a randomised controlled trial. British Journal of Psychiatry, 176:47-51.