Conventional guidelines for referee reports

How to write a good review (general conventional guidelines)

Some key points

• Cite evidence and reference specific parts of the research when giving feedback.

• Justify your critiques and claims in a reasoning-transparent way, rather than merely ‘"passing judgment." Avoid comments like "this does not pass the smell test".

• Provide specific, actionable feedback to the author where possible.

• Try to restate the authors’ arguments, clearly presenting the most reasonable interpretation of what they have written. See steelmanning.

• Be collegial and encouraging, but also rigorous. Criticize and question specific parts of the research without suggesting criticism of the researchers themselves.

We're happy for you to use whichever process and structure you feel comfortable with when writing your evaluation content.

One possible structure

Core

1. Briefly summarize the work in context

2. Highlight positive aspects of the paper and its strengths and contributions, considered in the context of existing research.

3. Most importantly: Identify and assess the paper's most important and impactful claim(s). Are these supported by the evidence provided? Are the assumptions reasonable? Are the authors using appropriate methods?

4. Note major limitations and potential ways the work could be improved; where possible, reference methodological literature and discussion and work that models what you are suggesting.

Optional/desirable

• Offer suggestions for increasing the impact of the work, for incorporating the work into global priorities research and impact evaluations, and for supporting and enhancing future work.

• Discuss minor flaws and their potential revisions.

• Desirable: formal 'claim identification and assessment'

Please don't spend time copyediting the work. If you like, you can give a few specific suggestions and then suggest that the author look to make other changes along these lines.

Remember: The Unjournal doesn’t “publish” and doesn’t “accept or reject.” So don’t give an Accept, Revise-and-Resubmit', or Reject-type recommendation. We ask for quantitative metrics, written feedback, and expert discussion of the validity of the paper's main claims, methods, and assumptions.

Writing referee reports: resources and benchmarks

Economics How to Write an Effective Referee Report and Improve the Scientific Review Process (Berk et al, 2017)

Semi-relevant: Econometric Society: Guidelines for referees

Report: Improving Peer Review in Economics: Stocktaking and Proposal (Charness et al 2022)

Open Science

PLOS (Conventional but open access; simple and brief)

Peer Community In... Questionnaire (Open-science-aligned; perhaps less detail-oriented than we are aiming for)

Open Reviewers Reviewer Guide (Journal-independent “PREreview”; detailed; targets ECRs)

General, other fields

The Wiley Online Library (Conventional; general)

"Peer review in the life sciences (Fraser)" (extensive resources; only some of this is applicable to economics and social science)

Other templates and tools

Collaborative template: RRR assessment peer review

Introducing Structured PREreviews on PREreview.org

‘the 4 validities’ and seaboat