"Direct evaluation" track
Proposal and discussion of this here
Second track: Direct evaluation of prominent work
We have added an "additional track" for commissioning research to be evaluated by The Unjournal:
Choose a set of "top-tier working paper series" as well as medium-to-top-tier journals.
Program started with the NBER working paper series.
July 26 2023: We have expanded this beyond NBER to research posted in other exclusive working paper archives and to work where all authors seem to be prominent/secure/established. See Direct evaluation: eligibility rules and guidelines.
Identify relevant papers in this series, according to criteria outlined and linked above (i.e., relevance, strength, need for further review). For NBER this tends to include
recently released work in the early stages of the journal peer-review process, particularly if it addresses a timely subject; as well as
work that has been around for many years, is widely cited and influential, yet has never been published in a peer-reviewed journal.
We aim to do this systematically and transparently, so authors do not feel singled out, nor left out.
Notify the work's authors that The Unjournal plans to commission evaluations. Not asking for permission, but
making them aware of The Unjournal, the process, the benefits to authors, and their opportunities to engage with the evaluation and publicly respond to the evaluation before it is made public;
letting the authors complete our forms if they wish, giving some further information about the paper or project and, for example, adding a "permalink" to updated versions;
asking if there are authors in sensitive career positions justifying a temporary "embargo"; and
asking the authors if there is specific feedback they would like to receive.
Reaching out to and commissioning evaluators, as in our regular process. Considerations:
Evaluators should be made aware that the authors have not directly requested this review, but have been informed it is happening. We suspect this will tend to drive evaluators to prefer to remain anonymous (but we will not push this as a norm).
As this will allow us to consider a larger set of papers more quickly, we can reach out to multiple evaluators more efficiently.
Aside: in the future, we hope to work directly with working paper series, associations, and research groups to get their approval and engagement with Unjournal evaluations. We hope that having a large share of papers in your series evaluated will serve as a measure of confidence in your research quality. If you are involved in such a group and are interested in this, please reach out to us (theunjournal@gmail.com).
Choosing NBER papers: steps taken (internal notes here, though these may be out of date; Airtable may be more relevant)
Direct evaluation: eligibility rules and guidelines
NBER
All NBER working papers are generally eligible, but watch for exceptions where authors seem vulnerable in their career. (And remember, we contact authors, so they can plead their case.)
CEPR
We treat these on a case-by-case basis and use discretion. All CEPR members are reasonably secure and successful, but their co-authors might not be, especially if these co-authors are PhD students they are supervising.
Papers or projects posted in any other working paper (pre-print) series
These are eligible (without author permission) if all authors
have tenured or ‘long term’ positions at well-known, respected universities or other research institutions, or
have tenure-track positions at top universities (e.g., top-20 globally by some credible rankings), or
are clearly not pursuing an academic career (e.g., the "partner at the aid agency running the trial").
On the other hand, if one or more authors is a PhD student close to graduation or an untenured academic outside a "top global program,’’ then we will ask for permission and potentially offer an embargo.
A possible exception to this exception: If the PhD student or untenured academic is otherwise clearly extremely high-performing by conventional metrics; e.g., an REStud "tourist" or someone with multiple published papers in top-5 journals. In such cases the paper might be considered eligible for direct evaluation.
Note: The above involves a substantial discretionary element. We aim to operationalize the above into better-defined rules where this is feasible. This is important to cultivate hands-off fairness and avoid favoritism.
Last updated