Comment on page

'Direct evaluation' track

Proposal and discussion of this here
2 Nov 2022 update: Our management team has decided to pilot the 'second track' specifically for NBER working papers.

Second track: Direct evaluation of prominent work

We are adding the following 'additional track' for commissioning research to be evaluated by The Unjournal,
  1. 1.
    Choose a set of 'top tier working paper series’ as well as medium-to-top tier journals
    • Starting with the NBER working paper series​
    • We may expand this to, e.g., work published in the Annual Review of Psychology
    ​
  2. 2.
    Identify relevant papers in this series, according to criteria outlined and linked above (relevance, strength, need for further review).
    For NBER this is likely to include...
    • recently released work in the early stages of the journal peer-review process, particularly if it addresses a timely subject as well as
    • work that has been around for many years, been widely cited and influential, yet never published in a peer-reviewed journal.
    We will try to do this in as systematically and transparently as feasible, so authors do not feel 'singled out'
  3. 3.
    Notify the work's authors that The Unjournal plans to commission evaluations. Not 'asking for permission' but...
    • Making them aware of The Unjournal, the process, the benefits to authors, and their opportunities to engage with the evaluation, and publicly respond to the evaluation before it is made public
    • Letting the authors complete our forms, if they wish to, giving some further information about the paper/project and, e.g., adding a 'permalink' to updated versions
    • Asking if there are authors in sensitive career positions justifying a temporary 'embargo',
    • Asking the authors if there is specific feedback they would like to receive
    ​
  4. 4.
    Reaching out to and commissioning evaluators, as in our regular process. Considerations:
    • These evaluators will be notified that the authors have not directly requested this review but they are informed it is happening. We suspect this will tend to drive evaluators to prefer to remaining anonymous.
    • As this will allow us to consider a larger set of papers more quickly, we can reach out to multiple evaluators more efficiently.

The case for such a (proposed) pivot:

  1. 1.
    Public benefit: Working papers (especially NBER) are already influencing policy and debate; yet they have not been peer-reviewed, and may take years to go through this process, if ever (e.g., many NBER papers are never published in peer-reviewed journals). However, it is difficult to understand their limitations unless you happen to have attended an academic seminar where they have been presented. Evaluating these publicly will provide a service.
  2. 2.
    'Fear of public evaluation/Safety in numbers': There may be some shyness or reluctance to participate in The Unjournal evaluation process (see our benefits to authors discussion). It is scary to be a first mover, and it may feel unfair to be among the few people to have an evaluation of your work out there in public (in spite of the Bayesian arguments presented in the previous link). There should be 'safety' in numbers: having a substantial number of prominent papers publicly evaluated by The Unjournal will ease this concern.
  3. 3.
    'Passive evaluation may be preferred to active consent': Academics (especially early-career) may also worry that they will seem weird or rebellious by submitting to The Unjournal and that this may be taken as 'rejecting mainstream system norms'. Again, this will be less of a problem if a substantial number of public evaluations of prominent papers are out there. You will be 'in good company'. Furthermore, if we are simply identifying papers for evaluation, the authors of these papers cannot be seen as 'rebelling' or 'being weird' (as they did not choose this).
  4. 4.
    Piloting and building a track record/demonstration: The Unjournal needs a reasonably large set of high-quality relevant work to evaluate, in order to help us build our system and improve our processes. Putting out a body of curated evaluation work will also allow us to demonstrate the reasonableness and reliability of this process.

Discussion: possible downsides and risks from this proposed change

  1. 1.
    Negative backlash: Some authors may dislike having their work publicly evaluated, particularly when there is substantial criticism. Academics complain a lot about unfair peer reviews, but the difference is that here the evaluations will be made public. This might lead The Unjournal to be the target of some criticism.
Responses:
  • Public engagement in prominent and influential work is fair and healthy. It is good to promote public intellectual debate. Of course, this needs to allow constructive criticism as well as informative praise.
  • We will work to ensure that the evaluations we publish involve constructive dialogue, avoid unnecessary harshness, and provide reasons for their critiques. We also give authors the opportunity to respond.
  • We are focusing on more prominent papers, with authors in more secure positions; and we offer a potential 'embargo' for sensitive career situations
2. Less author engagement: If authors do not specifically choose to have their work evaluated, they are likely to engage less with the process.
Response: This is something we will keep an eye on, considering the benefits against the costs.
​
3. Evaluator/referee reluctance: As noted above, evaluators may be more reluctant to provide ratings and feedback on work where the author has not instigated the process.
Response: This should basically be addressed by the fact that we allow evaluators to remain anonymous. A potential cost here is discouraging signed evaluations, which themselves have some benefits (as well as possible costs).
​
4. Slippery-slope towards 'unfairly reviewing work too early': In some fields, working papers are released at a point where the author does not wish them to be evaluated, and is not implicitly making strong claims about the validity of this work. In economics, working papers tend to be released when they are fairly polished and the authors do in fact seek feedback and citations; the NBER series is a particularly prominent example. However, if we are not careful one might be concerned that we will extend the scope of 'direct evaluation' too far.
Response: We will be careful with this. Initially, we are extending this only to the NBER series. Next, we may consider direct evaluation of fairly prestigious publications in 'actual peer-reviewed journals', particularly in fields (such as psychology) where the peer review process is much faster than in economics. As NBER is basically "USA only", we may also consider extending this to other series such as CEPR, but being careful about the prestige/vulnerability tradeoffs.
​
Aside: in the future, we hope to work directly with working paper series, associations, and research groups to get their approval and engagement with The Unjournal evaluation. We hope that "having a large share of papers in your series evaluated" will serve as a measure of confidence in your research quality. If you are involved in such a group and are interested in this, please reach out to us ([email protected]).
Choosing NBER papers: steps taken
1 Dec 2022 update
  • Further feedback and suggestions from across The Unjournal management team
  • Assigned 'assessors' to prioritize these
  • About five NBER papers (so far), selected for direct evaluation, contacting authors and referees
4 Nov update:
  • Already-added papers from other sources (syllabi etc)
  • Searched Rethink Priorities Zotero
  • EA Forum: Searched ‘nber’ with ‘advanced search’; Went through first 4 pages or so (by karma and recent date), added papers that seemed meaningful (paper is relevant, NBER but not published in a prominent journal)
  • Same but just for “Past Year” … did the first 4 pages of this
  • Searched NBER WP series for “animal welfare”
  • Captured the 'Topics' (NBER label) of all papers above for focusing our later search
​
​