Formats, research stage, publication status

Formats

We will consider research in almost any format — it does not need to be a frozen pdf or even a linear paper: see Promoting 'Dynamic Documents' and 'Living Research Projects'

Accessibility

We will only evaluate research objects that are openly accessible without paywalls or logins. The NBER working paper series is a borderline case, as there are some limits to access, but we're covering it for now because of it's importance and because there are workarounds to the access limits.

Publication/peer review status

For some fields, the peer review process is rapid and perhaps lower intensity; here 'post-peer-review evaluation' makes sense. In fields like economics, with very slow and high-intensity peer-review, we mostly consider evaluating work before it's been accepted in a peer-reviewed journal. I.e., 'working papers'. In such cases, our evaluations may contribute less to some parts of our value proposition (efficiency, author feedback and revision, early credibility signals). See .

But we sometimes consider (economics) papers already published in peer-reviewed journals, if we still see substantial value in having this work publicly evaluated and rated.

In particular, we may evaluate a 'journal-published paper' if

1. The paper significantly influences policy or funding decisions and therefore merits additional, transparent scrutiny.

2. Our evaluation process supports broader goals, such as fostering an impactful research community or refining our evaluation methods.

3. We have reasons to suspect that the journal's focus or field may be less likely to consider critical aspects of the paper, such as empirical robustness or policy relevance. E.g., a natural-science focused journal may neglect issues of causal inference that are more familiar to quantitative social scientists/economists.