Read the paper fully, making notes about what is clear and what is not.
Note technical issues, such as:
Are there experiments or analyses that should be done (or reframed)?
Do the authors describe new methods? If yes, how do these methods compare to existing methods?
Note stylistic issues, such as:
are the figures and accompanying legends clear?
Are acronyms used excessively and without justifiable reason?
Are there any missing or incorrect citations?
Composing the review
When formatting a first draft of a review, create three sections:
Write 3-5 sentences on the major goals of the paper
State the major successes of the paper.
“The major success of the paper is in developing a new model for side chain conformational heterogeneity”.
State the major weakness(es) of the paper (if any - could also be the major confusion you have with the way it is framed).
“The major weakness of the paper is that the model proposed is not tested”.
These can refer to the major points, which are elaborated below
Write 1-2 sentences on the place/impact of this paper in the field.
Enumerate only major issues that deal with data quality and data interpretation and reflect on the ability of the paper to accomplish it’s major goals
Each major issue should be its own paragraph or list of questions
this is your chance to ask both specific and open ended questions!
be specific and prescriptive, not passive agressive. Outline the experimental or textual revisions that are necessary to support the science.
DON’T: “it would be nice to include another technique.”
DO: “Your data do not fully support this specific (quote from paper) claim. Consider revising by either adding this specific control/technique described in this paper or altering the claim (paraphrase from paper) to allow for (reduced) alternative interpretation.”
REMEMBER: Most often, changes can be textual only! Do not suggest new experiments or controls just for the sake of suggesting something.
List all minor technical questions you have
It is important that you distinguish between things you think are done incorrectly/incompletely and things that are not explained clearly enough for a reader to understand.
List all minor stylistic issues
JF’s tend to focus a lot on clarity, but it’s great to have a reviewer who is eagle-eyed for all issues
Note any parts of the paper you do not feel like you are able to assess
“A significant part of the paper relies on a sophisticated analysis of mass spectrometry. I cannot offer expert feedback on the technical merits of this part of the paper.”
After composing the review
Reflect on the points you made and consider what is essential for supporting the arguments that are made in the paper
Avoid the instinct to focus on the gatekeeping role of review and instead try to reframe all comments around what will help the science the most
Re-read the review with an eye towards the reaction of the most junior author. Try to be empathetic to their point of view.