"Direct evaluation" track
Second track: Direct evaluation of prominent work
In addition to soliciting research submissions by authors, we directly prioritize unsubmitted research for evaluation, with a specific process and set of rules, outlined below.
Choose a set of "top-tier working paper series" and medium-to-top-tier journals.
This program started with the NBER working paper series. We expanded this beyond NBER to research posted in other exclusive working paper archives and to work where all authors seem to be prominent, secure, and established. See Direct evaluation: eligibility rules and guidelines.
Identify relevant papers in this series, following our stated criteria (i.e., relevance, strength, need for further review). For NBER this tends to include
recently released work in the early stages of the journal peer-review process, particularly if it addresses a timely subject; as well as
work that has been around for many years, is widely cited and influential, yet has never been published in a peer-reviewed journal.
We do this systematically and transparently; authors shouldn't feel singled out nor left out.
Notify the work's authors that The Unjournal plans to commission evaluations. We're not asking for permission, but
making them aware of The Unjournal, the process, the benefits to authors, and the authors' opportunities to engage with the evaluation and publicly respond to the evaluation before it is made public;
letting us know if we have the most recent version of the paper, and if updates are coming soon;
letting the authors complete our forms if they wish, giving further information about the paper or e.g. adding a "permalink" to updated versions;
asking if there are authors in sensitive career positions justifying a temporary "embargo"; and
asking the authors if there is specific feedback they would like to receive.
Reaching out to and commissioning evaluators, as in our regular process. Considerations:
Evaluators should be made aware that the authors have not directly requested this review, but have been informed it is happening.
As this will allow us to consider a larger set of papers more quickly, we can reach out to multiple evaluators more efficiently.
Aside: in the future, we hope to work directly with working paper series, associations, and research groups to get their approval and engagement with Unjournal evaluations. We hope that having a large share of papers in your series evaluated will serve as a measure of confidence in your research quality. If you are involved in such a group and are interested in this, please reach out to us (contact@unjournal.org).
Direct evaluation: eligibility rules and guidelines
NBER
All NBER working papers are generally eligible, but watch for exceptions where authors seem vulnerable in their career. (And remember, we contact authors, so they can plead their case.)
CEPR
We treat these on a case-by-case basis and use discretion. All CEPR members are reasonably secure and successful, but their co-authors might not be, especially if these co-authors are PhD students they are supervising.
Journal-published papers (i.e., 'post-publication evaluation')
In some areas and fields (e.g., psychology, animal product markets) the publication process is relatively rapid or it may fail to engage general expertise. In general, all papers that are already published in peer-reviewed journals are eligible for our direct track.
Papers or projects posted in any other working paper (pre-print) series
These are eligible (without author permission) if all authors
have tenured or ‘long term’ positions at well-known, respected universities or other research institutions, or
have tenure-track positions at top universities (e.g., top-20 globally by some credible rankings), or
are clearly not pursuing an academic career (e.g., the "partner at the aid agency running the trial").
On the other hand, if one or more authors is a PhD student close to graduation or an untenured academic outside a "top global program,’’ then we will ask for permission and potentially offer an embargo.
A possible exception to this exception: If the PhD student or untenured academic is otherwise clearly extremely high-performing by conventional metrics; e.g., an REStud "tourist" or someone with multiple published papers in top-5 journals. In such cases the paper might be considered eligible for direct evaluation.
Last updated