High-quality peer review is the essential backbone of any reputable scholarly journal and an important skill for researchers to acquire, yet few institutions formally teach students how to perform peer review. In place of formal training, some established researchers work with early career scientists to teach peer review under direct supervision. For example, an established researcher who has agreed to review a new submission with a journal will engage a junior faculty member to collaborate on the review. The reviewer comments will be submitted under the established researcher’s name along with a note to the editor about review contributions from the junior faculty member. However, this practice is not universal, even within different departments in the same institution, and may potentially be discouraged by journal policy.
At the journal level, most simply do not have sufficient resources to take on the task. While there are some notable exceptions such as the American Chemical Society’s ACS Reviewer Lab,1 online modules and general guidelines can only get a new reviewer so far, especially in niche and specialty fields.
With these challenges in mind, as the Managing Editor at the American Society for Clinical Pharmacology and Therapeutics, I established an interactive journal review club with the goal of providing hands-on peer-review experience and immediate feedback to participants. I decided to test the first session in person at our Society’s upcoming annual meeting. After getting approval from the Society’s senior leadership, I approached the Deputy Editor-in-Chief of our online-only, Open Access journal to ask if she would be interested in working on this project as the session facilitator. A one hour session was added to the meeting schedule and we developed the session agenda. Participants would be asked to review the first version of a submitted manuscript (that was ultimately accepted and published) and come to the session with their comments. The main focus of the session was the interaction between the participant-reviewers and authors. Which comments would be helpful to the authors and why? What types of comments are unhelpful or unrealistic?
Key People
A successful journal review club involves four distinct groups of people to ensure all perspectives of the journal’s peer-review process are represented. While someone might be invited as a participant, remember that she may also have a comment or question from the author perspective as well and should be encouraged to share.
- Facilitator. This volunteer should know the science being presented and understand the journal’s peer-review process and policies. An editor or editorial board member would be a natural option.
- Participants. Connecting to the intended audience is key. Look for existing relationships, such as members (for society journals), or previous authors or poor performing reviewers.
- Author(s). Successful peer review depends on the communication between reviewers and authors. An author’s perspective can be a valuable resource for those just starting with peer review.
- Editorial staff. A dedicated person to run the logistics of the event. As a bonus, staff can also help answer questions on journal specific policies or workflows.
How Does It Work?
The journal review club gives participants hands-on experience with peer review. About a week or two prior to joining the live session, each participant receives a simplified review form and the first version, including any supplemental materials, of a submitted manuscript. They are asked to review the manuscript and come to the session prepared to discuss found issues and provide a decision recommendation.
The live session starts with a brief overview of the day’s agenda. Then, one or two participants are asked to provide a brief summary of the paper to ensure everyone is on the same page. Participants are then encouraged to offer their comments on the paper, particularly regarding the science presented. The facilitator’s role at this point is to engage with participants and ask pointed questions to further the conversation. It’s also important to ask for the decision recommendations from all participants before moving on.
Next, participants receive (1) the original blinded reviewer comments the manuscript received, (2) the revised manuscript with changes tracked/highlighted, and (3) the author’s point-by-point response to the reviewer comments. The author(s) have the opportunity to address the comments from both the reviewers and club participants, highlighting what changes were made in the manuscript to address the comments or when there was disagreement and why.
Finally, the session ends with a discussion on peer review and how it helped strengthen the paper. Participants receive either a copy or link to the final, published version. This time is also a good opportunity to address questions participants (and authors!) have about the peer-review process from the editorial office. Some examples may include, how are editors/reviewers selected? Do I really need to pay attention to the guide to authors before I submit?
Logistics
To find a paper suitable for the journal review club, run a report of recently accepted or published research articles. Narrow your results by looking at the review history for each paper to ensure the reviewer comments were high quality. Consider further refining your results by looking at the authorship. Perhaps an author who is a member of the society or editorial board would make a good choice because of the preexisting relationship with the journal. Consider different time zones if the session will be held virtually, and avoid asking authors who would be expected to participate in the middle of the night.
Once a paper is selected, reach out to the first and/or corresponding author to ask if they would be willing to participate, then find a date and time for the session that would work for the authors, facilitator, and staff. After the session is scheduled, open registration to prospective participants. To ensure all participants are comfortable and have a chance to contribute to the conversation, set a target number of seats for the session that is not too large or small and remember to account for attrition when setting a cutoff point. The session can be promoted a number of ways: advertise on social media or society/journal newsletters, create a marketing email through the third-party service used by your company, or send personal email messages to key people. Responses can easily be tracked with a spreadsheet. Consider creating a calendar appointment with the location or connection details so participants are less likely to forget.
Collect all materials needed for the session. Create a simplified review form in a word processor, including brief instructions and a confidentiality note, and download the first version materials from the peer-review system. Compile these into a single PDF and send the file to participants a week or two before the live session. The original blinded reviewer comments the manuscript received, the revised manuscript with changes tracked/highlighted, and the author’s point-by-point response to the reviewer comments also need to be pulled from the peer-review system and readied for the session. It is important to note that any confidential comments from the original reviewers should remain confidential! While it takes a small amount of manual work, all materials and messaging are completed outside the peer-review system to avoid confusion with live papers.
Items to Consider
First, a great facilitator is essential to keep the discussion moving and on track. Sometimes participants can be shy when voicing their own comments and opinions, especially at the beginning, so the facilitator needs to be prepared with leading questions.
Second, find a manuscript that went through at least one round of revision and had some significant comments that needed to be addressed. If you choose a paper with minimal reviewer comments, be prepared to discuss how such papers should be handled.
Third, meet face to face, either in person or virtually via webcam, so it’s easier to engage with participants. This session can be successful at annual meetings and via video conferencing.
Fourth, limit the number of participants allowed. 10–12 seems to be an ideal number. It’s big enough that people don’t feel too shy about sharing their own thoughts and is small enough to allow everyone to participate fully within the time limit of the session.
Fifth, create a relaxed, comfortable atmosphere. The journal review club should be an enjoyable experience for everyone that allows for a truly engaging session.
Sixth, expect the unexpected. As with any postpublication peer review, there is a chance a mistake in the final paper might be discovered and need to be corrected. (It happened to me once, and the authors were grateful for the discovery so the paper could be corrected!)
Conclusion
With a little time and effort, any journal can engage early career scientists and help promote quality peer-review skills within their field. Engaged participants who come prepared with thoughtful comments and questions can be added to the reviewer pool, additionally strengthening their connection to the journal. This session can be easily adapted across disciplines and editorial offices of differing resources. Do not be afraid to try different approaches to the session; make it your own!
References and Links
Emma Shumeyko is currently the editorial systems manager at PNAS. This article was written while she was with the American Society for Clinical Pharmacology and Therapeutics.