"Direct evaluation" track
Last updated
Was this helpful?
Last updated
Was this helpful?
In addition to soliciting research by authors, we directly prioritize unsubmitted research for evaluation, with a specific process and set of rules, outlined below.
Choose a set of "top-tier working paper series" and medium-to-top-tier journals.
This program started with the . We expanded this beyond NBER to research posted in other exclusive working paper archives and to work where all authors seem to be prominent, secure, and established. See Direct evaluation: eligibility rules and guidelines.
Identify relevant papers in this series, following our stated criteria (i.e., , strength, ). For NBER this tends to include
recently released work in the early stages of the journal peer-review process, particularly if it addresses a timely subject; as well as
work that has been around for many years, is widely cited and influential, yet has never been published in a peer-reviewed journal.
We do this systematically and transparently; authors shouldn't feel singled out nor left out.
Notify the work's authors that The Unjournal plans to commission evaluations. We're not asking for permission, but
making them aware of The Unjournal, the process, the , and the authors' opportunities to engage with the evaluation and publicly respond to the evaluation before it is made public;
letting us know if we have the most recent version of the paper, and if updates are coming soon;
letting the authors complete our forms if they wish, giving further information about the paper or e.g. adding a "permalink" to updated versions;
asking if there are authors in sensitive career positions justifying a; and
asking the authors if there is specific feedback they would like to receive.
Reaching out to and commissioning evaluators, as in our regular process. Considerations:
Evaluators should be made aware that the authors have not directly requested this review, but have been informed it is happening.
As this will allow us to consider a larger set of papers more quickly, we can reach out to multiple evaluators more efficiently.
All NBER working papers are generally eligible, but watch for exceptions where authors seem vulnerable in their career. (And remember, we contact authors, so they can plead their case.)
We treat these on a case-by-case basis and use discretion. All CEPR members are reasonably secure and successful, but their co-authors might not be, especially if these co-authors are PhD students they are supervising.
In some areas and fields (e.g., psychology, animal product markets) the publication process is relatively rapid or it may fail to engage general expertise. In general, all papers that are already published in peer-reviewed journals are eligible for our direct track.
These are eligible (without author permission) if all authors
have tenured or ‘long term’ positions at well-known, respected universities or other research institutions, or
have tenure-track positions at top universities (e.g., top-20 globally by some credible rankings), or
are clearly not pursuing an academic career (e.g., the "partner at the aid agency running the trial").
On the other hand, if one or more authors is a PhD student close to graduation or an untenured academic outside a "top global program,’’ then we will ask for permission and potentially offer an embargo.
A possible exception to this exception: If the PhD student or untenured academic is otherwise clearly extremely high-performing by conventional metrics; e.g., an REStud "tourist" or someone with multiple published papers in top-5 journals. In such cases the paper might be considered eligible for direct evaluation.
Public benefit: Working papers (especially NBER) are already influencing policy and debate, yet they have not been peer-reviewed and may take years to go through this process, if ever (e.g., many NBER papers). However, it is difficult to understand the papers' limitations unless you happen to have attended an academic seminar where they were presented. Evaluating these publicly will provide a service.
Specifically for NBER: This working paper series is highly influential and relied upon by policy makers and policy journalists. It'd an elite outlet: only members of NBER are able to post working papers here.
Fear of public evaluation (safety in numbers): There may be some shyness or reluctance to participate in The Unjournal evaluation process (for reasons to do so, see our discussion). It is scary to be a first mover, and it may feel unfair to be among the few people to have an evaluation of your work out there in public (in spite of the Bayesian arguments presented in the previous link). There should be "safety" in numbers: having a substantial number of prominent papers publicly evaluated by The Unjournal will ease this concern.
Response: We will be careful with this. Initially, we are extending this evaluation process only to the NBER series. Next, we may consider direct evaluation of fairly prestigious publications in "actual" peer-reviewed journals, particularly in fields (such as psychology) where the peer-review process is much faster than in economics. As NBER is basically "USA-only", we have extended this to other series such as , while being sensitive to the prestige/vulnerability tradeoffs.
Aside: in the future, we hope to work directly with working paper series, associations, and research groups to get their approval and engagement with Unjournal evaluations. We hope that having a large share of papers in your series evaluated will serve as a measure of confidence in your research quality. If you are involved in such a group and are interested in this, please reach out to us ().