Ask or search…

Conventional guidelines for referee reports

How to write a good review (general conventional guidelines)

Some general key points to consider
  • Cite evidence and reference specific parts of the research when giving feedback.
  • Try to justify your critiques and claims in a reasoning-transparent way, rather than merely ‘"passing judgment."
  • Provide specific, actionable feedback to the author where possible.
  • When considering the authors’ arguments, consider the most reasonable interpretation of what they have written (and state what that is, to help the author make their point more clearly). See steelmanning.
  • Be collegial and encouraging, but also rigorous. Criticize and question specific parts of the research without suggesting criticism of the researchers themselves.
We are happy for you to use whichever process and structure you feel comfortable with when writing a peer review.
One possible structure
  • Assign an overall score based on quantitative metrics (possible: brief discussion of these metrics).
  • Summarize the work and issues, and the research in context to convey your understanding and help others understand it.
  • Highlight positive aspects of the paper, strengths and contributions.
    • Assess the contribution of the work in context of existing research.
  • Note major limitations and potential ways the work could be improved; where possible, reference methodological literature and discussion and work that models what you are suggesting.
  • Discuss minor flaws and their potential revisions.
    • You are not obliged (or paid) to spend a great deal of time copyediting the work. If you like, you can give a few specific suggestions and then suggest that the author look to make other changes along these lines.
  • Offer suggestions for research agendas, increasing the impact of the work, incorporating the work into global priorities research and impact evaluations, and enhancing future work.
Remember: The Unjournal doesn’t “publish” and doesn’t “accept or reject.” So don’t give an Accept, Revise and Resubmit, or Reject-type recommendation. We just want quantitative metrics, some written feedback, and some relevant discussion.
"This paper is great, I would accept it without changes, what should I write/do?"
We still want your evaluation and ratings. Some things to consider as an evaluator in this situation:
  1. 1.
    We still want your quantitative ratings and predictions.
  2. 2.
    A paper or project is not only a good to be judged on a single scale. How useful is it, and to whom or what? We'd like you discuss its value in relation to previous work, it’s implications, what it suggests for research and practice, etc.
  3. 3.
    Even if the paper is great . . .
    • Would you accept it in the “top journal" in economics”? If not, why not?
    • Would you hire someone based on this paper?
    • Would you fund a major intervention (as a government policymaker, major philanthropist, etc.) based on this paper alone? If not, why not?
  4. 4.
    What are the most important and informative results of the paper?
  5. 5.
    Can you quantify your confidence in these "crucial" results, and their replicability and generalizability to other settings? Can you state your probabilistic bounds (confidence or credibility intervals) on the quantitative results (e.g., 80% bounds on QALYs/DALYs/or WELLBYs per $1,000).
  6. 6.
    Would any other robustness checks or further work have the potential to increase your confidence (narrow your belief bounds) in this result? Which?
  7. 7.
    Do the authors make it easy to reproduce the statistical (or other) results of the paper from shared data? Could they do more in this respect?
  8. 8.
    Communication: Did you understand all of the paper? Was it easy to read? Are there any parts that could have been better explained?
    • Is it communicated in a way that would make it useful to policymakers? Would it be useful to other researchers in this field, or in the general discipline?

Writing referee reports: resources and benchmarks

Open Science
PLOS (Conventional but open access; simple and brief)
Peer Community In... Questionnaire (Open-science-aligned; perhaps less detail-oriented than we are aiming for)
Open Reviewers Reviewer Guide (Journal-independent “pre-review”; detailed; targets ECRs)
The Wiley Online Library (Conventional; general)