← Go Back to the Impact Challenge Table of Contents
Peer review image licensed CC BY-SA
Another area in academia with untapped potential for demonstrating your impact is peer review.
New forms of peer review – open peer review for journals, post-publication peer review, and peer reviews written on sites like Publons – can help you establish expertise in your discipline. They turn anonymous service to your field into a standalone scholarly product and also communicate feedback on published work to your discipline, often more quickly.
Open peer review was borne of the idea that by making author and reviewer identities public, more civil and constructive peer reviews will be submitted, and peer reviews can be put into context.
And open post-publication peer review builds upon that by allowing anyone to publish a review of an already-published paper, whether on their blog or a standalone peer review platform like Faculty of 1000 or PubPeer. After all, why should official reviewers be the only ones allowed to share their views on a paper?We know there are concerns about open peer review. From Sarah Hare’s excellent article about open peer review:
“Throughout the course, I found that students were hesitant to embrace open peer review, even after we discussed the many flaws of closed peer review and the ways in which open peer review models address these limitations. Again and again, students equated open peer review with bias and a lack of rigor.”
Yet research is showing that making reviewers reports freely-readable doesn’t actually compromise the peer review process.
In this week’s challenge, we’ll explore your options for writing open peer reviews, talk about ways you can make your reviews citable and discoverable, and share tips for documenting your peer reviews on your CV. And we’ll provide some interesting testimonials along the way!
Traditional peer review
Publishers have long favored private, anonymous (blind) peer review, under the assumption that it would reduce bias and that authors would prefer for criticisms of their work to remain private. Turns out, their assumptions weren’t necessarily always backed up by evidence.
It can be easy for authors to guess the identities of their reviewers (especially in small fields). And yet, a consequence of this “anonymous” legacy system is that you, as a reviewer, can’t take credit for your work.Sure, you can say you’re a reviewer for Physical Review B, but you’re unable to point to specific reviews or discuss how your feedback made a difference. That means that others can’t read your reviews to understand your intellectual contributions to your field, which – in the case of some reviews – can be enormous.
Shades of open peer review
In recent years, scholars have increasingly called for an open alternative to traditional peer review. This has manifested in journals adopting open peer review (OPR), scholars taking to their blogs to review already-published work, and the proliferation of open and post-publication peer review sites like Faculty of 1000, PubPeer, and Publons.
Each shade of OPR has its advantages and disadvantages. Let’s take a closer look.
Open peer review for journals
Here’s how open peer review works, more or less: reviewers are assigned to a paper, and they know the author’s identity. They review the paper and sign their name. The reviews are then submitted to the editor and author (who now knows their reviewers’ identities, thanks to the signed reviews). When the paper is published, the signed reviews are published alongside it.
If you’d like to see one example, the journal PeerJ allows open peer reviews (after an initial single, blind review), as does the psychology-focused journal, Meta-Psychology.
Participating in journal-based OPR can be a good way to experiment with OPR as the author, journal, and reviewer alike officially sanction it.
One drawback to this type of Open Peer Review is that some journals do not provide permanent identifiers (like DOIs) for the reviews themselves, making it difficult to track the reach and impact of your review rather than for the journal article you’ve reviewed. Luckily, PeerJ has changed that. They issue DOIs for open peer reviews, which comprise 40% of their reviews.
Third-party open and post-publication peer review sites
In the past few years, a number of standalone, independent peer review sites have emerged: PubPeer, Publons, and Faculty of 1000 are among the many. These sites allow you to review both published and under-review papers on their platform, and in the case of Publons, export your reviews to journals for use.
These sites also allow you to submit your reviews as open peer reviews, and to create profiles showcasing your peer reviews. Some sites like Publons also issue DOIs for reviews, making them citable research objects.
Blogging as open post-publication peer review
In this type of open peer review, scholars take to their blogs to share their thoughts on a recently published paper or preprint. These reviews can run the gamut from highly technical reviews oriented towards other academics (a good example is this post on Rosie Redfield’s blog) to reviews written for a more general audience (like Mike Eisen’s post on the same study).
A major advantage to blogging your open peer reviews is that you don’t have to have permission to do it; you can just fire up your blog and start reviewing. But a downside is that the review isn’t formally sanctioned by the journal, so it can carry less weight than formal reviews.
No matter what type of open peer review you opt for, if it’s got your name attached to it and is available for all to read, you can use it to showcase your expertise in your area of research.
Write an Open Peer Review
If you’d prefer to go the journal-sanctioned open peer review route, choose to review for journals that already offer Open Peer Review. A number of journals allow it (BMJ, PeerJ, and F1000 Research, among others).
To find others, head to the Directory of Open Access Journals (DOAJ) and follow these instructions:
- On DOAJ’s home page click the “Advanced Search” link underneath the search box
- On the facets down the left side, click on “Journals” underneath the “Journals and Articles” facet. The facets will expand once you do this.
- Click on the “Peer Review” facet (second from the bottom)
- Click on the “Open Peer Review” facet
- You can now see the open access journals that support open peer review. At the time of this writing, there were 137 such journals. You may want to enter a disciplinary keyword (e.g. “psychiatry” or “geography”) to narrow this to your field.
If you are an author in a STEM field you can also try a similar search using Cofactor’s journal selector tool:
- Head to the Cofactor journal selector tool
- Click “Peer review,”
- Select “Fully Open” in the middle column
- Click “Search” to see a full list of Open Peer Review journals.
Alternatively, you can write your peer review on a stand-alone post-publication peer review platform like Publons, or others we mentioned above. Find a platform that works for you, sign up for it (most platforms require you to set up an account), and start reviewing.
And if you choose to do open post-publication peer review through your blog, it’s even easier – just log on and start writing!
Get citations and altmetrics for your peer reviews
Once your open peer reviews are online, you can discover citations, shares, discussions, and bookmarks of them if they’ve got permanent identifiers that are easily trackable. The most common ID that’s used for peer reviews is a DOI.
There are two main ways you can get a DOI for your reviews:
Review for a journal like PeerJ or peer review platform like Publons that issues DOIs automatically
Archive your review in a repository that issues DOIs, like Figshare
When you’ve got your DOI, use it! Include it on your CV (more on that below), as a link when sharing your reviews with others, and so on. And encourage others to always link to your review using the DOI resolver link (these are created by putting “http://doi.org/” in front of your DOI; here’s an example of what one looks like: http://doi.org/10.7287/peerj.603v0.1/reviews/2).
Elevate your peer reviews
Peer review may be viewed primarily as a “service” activity, but things are changing – and you can help change them even more quickly. Here’s how.
As a reviewer, raise awareness by listing and linking to your journal-sanctioned reviews on your CV, adjacent to any mentions of the journals you review for. By linking to your specific reviews (using the DOI resolver link we talked about above), anyone looking at your CV can easily read the reviews themselves.
You can also illustrate for others the impacts of open peer review by including citations and altmetrics for your reviews on your CV. An easy way to do that is to include on your CV a link to the review on your Impactstory profile. You can also include other quantitative measures of your reviews’ quality, like Peerage of Science’s Peerage Essay Quality scores or a number of other quantitative indicators of peer-review quality. Just be sure to provide context to any numbers you include.
If you decide to do open peer reviews mostly on your blog or standalone peer review sites, you’ll likely not want to list them under Service to Journals, per se, but instead perhaps under Outreach or more general Service to your field.
Limitations
A big concern for early career researchers and graduate students lies in openly criticizing senior researchers in their field. What if they’re retaliated against? Anonymity would protect these ECR-reviewers from their colleagues.
Yet as Mick Watson argues, any retaliation that could theoretically occur would be considered a form of scientific misconduct, on par with plagiarism and therefore off-limits to scholars.
You are the best judge of whether or not a peer review could have unintended consequences, and we suggest that you use your good judgment when deciding to make your review open or not.
Homework
Your assignment for today is to choose an article to review on your blog. If you’re new to reviewing or unsure how to go about writing a free-form peer review as a blog post, here are some guides to help you get started.
If you’re still unconvinced about the benefits of open peer review, here are some resources for you to take a look at:
- Lee, C.J., Sugimoto, C.R., Zhang, G., & Cronin, B. ( 2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2 – 17. DOI: 10.1002/asi.22784. (may be behind a paywall; access available via OU Libraries)
- Ford, E. (2013). Defining and characterizing open peer review. Journal of Scholarly Publishing, 44(4), 311 – 326. https://doi.org/10.3138/jsp.44-4-001. Post print available at: https://pdxscholar.library.pdx.edu/cgi/viewcontent.cgi?article=1000&context=ulib_fac
- Maharg, P. & Duncan, N. (2007). Black Box, Pandora's Box or Virtual Toolbox? An Experiment in a Journal's Transparent Peer Review on the Web, International Review of Law, Computers & Technology, 21(2), 109-128, DOI: 10.1080/13600860701492104. Post print available at: http://openaccess.city.ac.uk/4343/1/Maharg-Duncan%20page%20proofs.pdf
- Denson, S. (2017). Open Peer-Review as Multimodal Scholarship. Cinema Journal, 56(4), 141-143. University of Texas Press. Retrieved April 1 from Project MUSE database.