"Direct evaluation" track

Proposal and discussion of this here

Second track: Direct evaluation of prominent work

In addition to soliciting research submissions by authors, we directly prioritize unsubmitted research for evaluation, with a specific process and set of rules, outlined below.

1. Choose a set of "top-tier working paper series" and medium-to-top-tier journals.

   This program started with the NBER working paper series. We expanded this beyond NBER to research posted in other exclusive working paper archives and to work where all authors seem to be prominent, secure, and established. See Direct evaluation: eligibility rules and guidelines.

2. Identify relevant papers in this series,following our stated criteria (i.e., relevance, strength, need for further review). For NBER this tends to include

   • recently released work in the early stages of the journal peer-review process, particularly if it addresses a timely subject; as well as

   • work that has been around for many years, is widely cited and influential, yet has never been published in a peer-reviewed journal.

We do this systematically and transparently; authors shouldn't feel singled out nor left out.

3. Notify the work's authors that The Unjournal plans to commission evaluations. We're not asking for permission, but

   • making them aware of The Unjournal, the process, the benefits to authors, and the authors' opportunities to engage with the evaluation and publicly respond to the evaluation before it is made public;

   • letting us know if we have the most recent version of the paper, and if updates are coming soon;

   • letting the authors complete our forms if they wish, giving further information about the paper or e.g. adding a "permalink" to updated versions;

   • asking if there are authors in sensitive career positions justifying a temporary "embargo"; and

   • asking the authors if there is specific feedback they would like to receive.

4. Reaching out to and commissioning evaluators, as in our regular process. Considerations:

   • Evaluators should be made aware that the authors have not directly requested this review, but have been informed it is happening.

   • As this will allow us to consider a larger set of papers more quickly, we can reach out to multiple evaluators more efficiently.

The case for this "direct evaluation"

1. Public benefit: Working papers (especially NBER) are already influencing policy and debate, yet they have not been peer-reviewed and may take years to go through this process, if ever (e.g., many NBER papers are never published in peer-reviewed journals). However, it is difficult to understand the papers' limitations unless you happen to have attended an academic seminar where they were presented. Evaluating these publicly will provide a service.

   • Specifically for NBER: This working paper series is highly influential and relied upon by policy makers and policy journalists. It'd an elite outlet: only members of NBER are able to post working papers here. Membership is prestigious and available only by invitation.

2. Fear of public evaluation (safety in numbers): There may be some shyness or reluctance to participate in The Unjournal evaluation process (for reasons to do so, see our benefits to authors discussion). It is scary to be a first mover, and it may feel unfair to be among the few people to have an evaluation of your work out there in public (in spite of the Bayesian arguments presented in the previous link). There should be "safety" in numbers: having a substantial number of prominent papers publicly evaluated by The Unjournal will ease this concern.

3. Passive evaluation may be preferred to active consent: Academics (especially early-career) may also worry that they will seem weird or rebellious by submitting to The Unjournal, as this may be taken as "rejecting mainstream system norms." Again, this will be less of a problem if a substantial number of public evaluations of prominent papers are posted. You will be in good company. Furthermore, if we are simply identifying papers for evaluation, the authors of these papers cannot be seen as rejecting the mainstream path (as they did not choose to submit).

4. Piloting and building a track record or demonstration: The Unjournal needs a reasonably large set of high-quality, relevant work to evaluate in order to help us build our system and improve our processes. Putting out a body of curated evaluation work will also allow us to demonstrate the reasonableness and reliability of this process.

Discussion: possible downsides and risks from this, responses

1. Negative backlash: Some authors may dislike having their work publicly evaluated, particularly when there is substantial criticism. Academics complain a lot about unfair peer reviews, but the difference is that here the evaluations will be made public. This might lead The Unjournal to be the target of some criticism.

Responses:

• Public engagement in prominent and influential work is fair and healthy. It is good to promote public intellectual debate. Of course, this process needs to allow constructive criticism as well as informative praise.

• We will work to ensure that the evaluations we publish involve constructive dialogue, avoid unnecessary harshness, and provide reasons for their critiques. We also give authors the opportunity to respond.

• We are focusing on more prominent papers, with authors in more secure positions. Additionally, we offer a potential "embargo" for sensitive career situations, e.g., those that might face early-career researchers.

2. Less author engagement: If authors do not specifically choose to have their work evaluated, they are less likely to engage fullly with the process.

Response: This is something we will keep an eye on, weighing the benefits and costs.

3. Evaluator/referee reluctance: As noted above, evaluators may be more reluctant to provide ratings and feedback on work where the author has not instigated the process.

Response: This should largely be addressed by the fact that we allow evaluators to remain anonymous. A potential cost here is discouraging signed evaluations, which themselves have some benefits (as well as possible costs).

4. Slippery-slope towards "unfairly reviewing work too early": In some fields, working papers are released at a point where the author does not wish them to be evaluated, and where the author is not implicitly making strong claims about the validity of this work. In economics, working papers tend to be released when they are fairly polished and the authors typically seek feedback and citations. The NBER series is a particularly prominent example. However, we don't want extend the scope of direct evaluation too far.

Response: We will be careful with this. Initially, we are extending this evaluation process only to the NBER series. Next, we may consider direct evaluation of fairly prestigious publications in "actual" peer-reviewed journals, particularly in fields (such as psychology) where the peer-review process is much faster than in economics. As NBER is basically "USA-only", we have extended this to other series such as CEPR, while being sensitive to the prestige/vulnerability tradeoffs.

Aside: in the future, we hope to work directly with working paper series, associations, and research groups to get their approval and engagement with Unjournal evaluations. We hope that having a large share of papers in your series evaluated will serve as a measure of confidence in your research quality. If you are involved in such a group and are interested in this, please reach out to us ([e-mail]).

Direct evaluation: eligibility rules and guidelines

NBER

All NBER working papers are generally eligible, but watch for exceptions where authors seem vulnerable in their career. (And remember, we contact authors, so they can plead their case.)

CEPR

We treat these on a case-by-case basis and use discretion. All CEPR members are reasonably secure and successful, but their co-authors might not be, especially if these co-authors are PhD students they are supervising.

Papers or projects posted in any other working paper (pre-print) series

These are eligible (without author permission) if all authors

• have tenured or ‘long term’ positions at well-known, respected universities or other research institutions, or

• have tenure-track positions at top universities (e.g., top-20 globally by some credible rankings), or

• are clearly not pursuing an academic career (e.g., the "partner at the aid agency running the trial").

On the other hand, if one or more authors is a PhD student close to graduation or an untenured academic outside a "top global program,’’ then we will ask for permission and potentially offer an embargo.

• A possible exception to this exception: If the PhD student or untenured academic is otherwise clearly extremely high-performing by conventional metrics; e.g., an REStud "tourist" or someone with multiple published papers in top-5 journals. In such cases the paper might be considered eligible for direct evaluation.