"Direct evaluation" track

Direct evaluation of prominent work

In addition to soliciting research submissions by authors, we have a process for sourcing and prioritizing unsubmitted research for evaluation. For some of this research we ask for author engagement but do not require their permission.

For this "direct evaluation" process we generally ...

1. Choose a set of "top-tier working paper series" and medium-to-top-tier journals, as well as research posted in other exclusive working paper archives and to work where all authors seem to be prominent, secure, and established. See "Direct evaluation" track.

2. Identify relevant papers in this series, following our stated criteria (i.e., relevance, strength, need for further review). For NBER this tends to include

   • recently released work in the early stages of the journal peer-review process, particularly if it addresses a timely subject; as well as

   • work that has been around for many years, is widely cited and influential, yet has never been published in a peer-reviewed journal.

We do this systematically and transparently; authors shouldn't feel singled out nor left out.

1. Notify the work's authors that The Unjournal plans to commission evaluations. We're not asking for permission, but

   • making them aware of The Unjournal, the process, the benefits to authors, and the authors' opportunities to engage with the evaluation and publicly respond to the evaluation before it is made public;

   • letting us know if we have the most recent version of the paper, and if updates are coming soon;

   • letting the authors complete our forms if they wish, giving further information about the paper or e.g. adding a "permalink" to updated versions;

   • asking if there are authors in sensitive career positions justifying a temporary "embargo"; and

   • asking the authors if there is specific feedback they would like to receive.

2. Reaching out to and commissioning evaluators, as in our regular process. Considerations:

   • Evaluators should be made aware that the authors have not directly requested this review, but have been informed it is happening.

   • As this will allow us to consider a larger set of papers more quickly, we can reach out to multiple evaluators more efficiently.

The case for this "direct evaluation"

1. Public benefit: Working papers (especially NBER) are already influencing policy and debate, yet they have not been peer-reviewed and may take years to go through this process, if ever (e.g., many NBER papers are never published in peer-reviewed journals). However, it is difficult to understand the papers' limitations unless you happen to have attended an academic seminar where they were presented. Evaluating these publicly will provide a service.

   • Specifically for NBER: This working paper series is highly influential and relied upon by policy makers and policy journalists. It'd an elite outlet: only members of NBER are able to post working papers here. Membership is prestigious and available only by invitation.

2. Fear of public evaluation (safety in numbers): There may be some shyness or reluctance to participate in The Unjournal evaluation process (for reasons to do so, see our benefits to authors discussion). It is scary to be a first mover, and it may feel unfair to be among the few people to have an evaluation of your work out there in public (in spite of the Bayesian arguments presented in the previous link). There should be "safety" in numbers: having a substantial number of prominent papers publicly evaluated by The Unjournal will ease this concern.

3. Passive evaluation may be preferred to active consent: Academics (especially early-career) may also worry that they will seem weird or rebellious by submitting to The Unjournal, as this may be taken as "rejecting mainstream system norms." Again, this will be less of a problem if a substantial number of public evaluations of prominent papers are posted. You will be in good company. Furthermore, if we are simply identifying papers for evaluation, the authors of these papers cannot be seen as rejecting the mainstream path (as they did not choose to submit).

4. Piloting and building a track record or demonstration: The Unjournal needs a reasonably large set of high-quality, relevant work to evaluate in order to help us build our system and improve our processes. Putting out a body of curated evaluation work will also allow us to demonstrate the reasonableness and reliability of this process.

Discussion: possible downsides and risks from this, responses

1. Negative backlash: Some authors may dislike having their work publicly evaluated, particularly when there is substantial criticism. Academics complain a lot about unfair peer reviews, but the difference is that here the evaluations will be made public. This might lead The Unjournal to be the target of some criticism.

Responses:

• Public engagement in prominent and influential work is fair and healthy. It is good to promote public intellectual debate. Of course, this process needs to allow constructive criticism as well as informative praise.

• We will work to ensure that the evaluations we publish involve constructive dialogue, avoid unnecessary harshness, and provide reasons for their critiques. We also give authors the opportunity to respond.

• We are focusing on more prominent papers, with authors in more secure positions. Additionally, we offer a potential "embargo" for sensitive career situations, e.g., those that might face early-career researchers.

2. Less author engagement: If authors do not specifically choose to have their work evaluated, they are less likely to engage fullly with the process.

Response: This is something we will keep an eye on, weighing the benefits and costs.

3. Evaluator/referee reluctance: As noted above, evaluators may be more reluctant to provide ratings and feedback on work where the author has not instigated the process.

Response: This should largely be addressed by the fact that we allow evaluators to remain anonymous. A potential cost here is discouraging signed evaluations, which themselves have some benefits (as well as possible costs).

4. Slippery-slope towards "unfairly reviewing work too early": In some fields, working papers are released at a point where the author does not wish them to be evaluated, and where the author is not implicitly making strong claims about the validity of this work. In economics, working papers tend to be released when they are fairly polished and the authors typically seek feedback and citations. The NBER series is a particularly prominent example.

Response: We will be careful with this. Initially, we started this 'direct evaluation' process only for the NBER series. We later extended this, as noted above.

Aside: in the future, we hope to work directly with working paper series, associations, and research groups to get their approval and engagement with Unjournal evaluations. We hope that having a large share of papers in your series evaluated will serve as a measure of confidence in your research quality. If you are involved in such a group and are interested in this, please reach out to us ([e-mail]).

Dec 7, 2024: We have updated some of our rules and guidelines on this page. These will be applied going forward (in future contacts with authors) but not retroactively.

Direct evaluation of "prestige work": eligibility rules & guidelines

NBER

All NBER working papers are generally eligible, with .

CEPR

We treat these on a case-by-case basis and use discretion. All CEPR members are reasonably secure and successful, but their co-authors might not be, especially if these co-authors are PhD students they are supervising.

Journal-published papers (i.e., 'post-publication evaluation')

In some areas and fields (e.g., psychology, animal product markets) the publication process is relatively rapid or it may fail to engage general expertise. In general, all papers that are already published in peer-reviewed journals are eligible for our direct track.

(We discuss when we consider post-pub evaluation in Formats, research stage, publication status)

Papers or projects posted in any other working paper (pre-print) series

These are eligible (without author permission) if all authors/all lead authors "have high professional status" or are otherwise less career-sensitive to the consequences of this evaluation.

High professional status/less career-sensitive

We define this (at least for economics) as:

• having a tenured or ‘long term’ positions at well-known, respected universities or other research institutions, or

• having a tenure-track positions at a top universities (e.g., top-20 globally by some credible ranking) and having published one or more papers in a "top-five-equivalent" journal, or

• clearly not pursuing an academic career (e.g., the "partner at the aid agency running the trial").

On the other hand, if one or more authors is a PhD student or an untenured academic outside a "top global program,’’ then we will ask for permission and potentially offer an embargo.

Exceptions to these exceptions

If the PhD student or untenured academic is otherwise clearly extremely high-performing by conventional metrics; e.g., an REStud "tourist" or someone with multiple published papers in top-5 journals. In such cases the paper might be considered eligible for direct evaluation.

See also Direct evaluation of "impactful work"

Under review/R&R at a journal? The fact that the paper is under submission or in "revise and resubmit" at a top journal does not preclude us from evaluating it. In some cases it may be particular important and helpful to evaluate work at this stage. But we'd like to be aware of this, as it can weigh into our considerations and timing.

Direct evaluation of "impactful work"

We will also evaluate work directly, without requiring author permission, where it is clear that this research is already influencing a substantial amount of funding in impact-relevant areas, or substantially influencing policy considerations. Much of this work will be evaluated as part of our "Applied and Policy" Track.