Comment on page
Evaluation manager process
See Mapping evaluation workflowfor an updated careful overview and flowchart. The material below overlaps that somewhat.
Update 28 November, 2023: We now have an 'alpha version' of a PubPub V7 setup that will serve as an automated "editorial management tool". We are piloting this for a few papers and aim to fully transition soon. Reach out if you are interested in trying this.
You should have been registered as the Evaluation Manager in the Airtable
crucial_rsx table. You may find other useful information and links there, including a potential link to "Bespoke Evaluation Notes" in the field link_discussion_what_to_evaluate ; you may want to consult and add to these.- 1.If the work falls under our "Direct evaluation" track (NBER+; should be noted in the Airtable), please write to the authors of the paper to let them know that their paper will be evaluated by The Unjournal and ask them about the status of their paper. In particular, we want to know if the version we are considering is the most recent version.
- See an example letter template for this HERE. Other templates (for this and for other correspondence) are found in our Airtable
text_templatestable. - Please wait 5 business days to give the authors a chance to respond before directly engaging evaluators.
- 2.Submitted work: Otherwise, if the authors submitted their work to The Unjournal directly, we need to inform them that their work was selected (and make sure all authors are OK with this).
- 3.Other selected work: Otherwise, if the paper was selected (not submitted), and it doesn't qualify for the Direct Evaluation track (e.g., it is not NBER and/or it has early-career researcher authors) we need to ask the authors' permission to proceed (according to our policies), and of course we also want them to notify us about the paper's status, updates, etc.
The Unjournal may have already done some of the above (before you were assigned as the Evaluation Manager); please check if this is the case.
Please use the link below to help you consider how to choose evaluators. Normally we commission two evaluators; sometimes we for a third evaluation (e.g., where particular expertise is needed). In other cases (e.g., where the paper has become less relevant to evaluate), we may end the process after only a single evaluation.
Once you have chosen possible evaluators, reach out to them. A short personalized email seems to get the best response. If you like, you can use or adapt the text templates in the Airtable under the "evaluator" grouping. For example, the "referee outreach (pilot long)" or "short referee outreach (NBER phase)". You often will have to reach out to multiple potential evaluators as they may reject the offer, or not respond.
Update Nov 2023: We have a system on PubPub that mostly automates process of queueing invitations, reminders, etc. If you want to try out this system (in alpha) please reach out. See documentation/notes here.
Our policy is to send a reminder email five days after the original email. After ten days, if there is still no response, we move on to the next potential evaluator. You can use the "Referee informal followup/reminder" or "PS to referee request" text templates from the Airtable for these emails. (Nov. 2023: This and other reminders are being automated in our PubPub interface, which will be available soon)
To speed up the process and automate your reminders, you can use Boomerang, as explained in the following link:
If an evaluator agrees,
- Send them another email with the Google Doc with the evaluation template for them to complete,
- as well as a link to 'Bespoke Evaluation Notes' for this project, if it has been created (see template)
- Propose a deadline to complete the evaluation; we aim for a three-week turnaround, but we can be somewhat flexible. Make sure we agree on a date.
Nov. 2023: Within the new PubPub system there will be a direct link to the evaluation interface.
After the agreed time period has elapsed and they have not sent the evaluation, it helps to send a reminder email. This can be short and just direct them to the original email. Our current policy is to send an email on the due date and again five days later. (This will soon be automated in the PubPub system).
- 1.Check through the evaluation and ensure that nothing is missing from the evaluation; reach out to the evaluator for completions and clarifications if necessary.
- 2.If you have no further questions or amendments, you can direct the evaluator to payment information: Fiscal hosting and expenses
- 3.Wait to receive all the evaluations before proceeding to the next step.
- 1.Let the evaluators know that you will be sharing the evaluations with the authors. Inform them that their names will not be shared at this stage, but they are free to reach out to the authors. You could use the "letting evaluators know about sharing of evaluations with authors" text template in the Airtable.
- 2.Share the evaluations with the authors, removing any identifying information about the evaluator. You can compile the evaluations into a single document using our Google Docs template, which we can share with you. You can adapt or modify one of the responses in the Airtable under text_templates. Perhaps "share_eval_with_authors_simple" or "Compiled evaluations template -- for author sharing and publication".
- 3.If the authors say "we won't be able to respond," then let them know we will put it up shortly and proceed. Otherwise we give them 2 weeks to respond before putting it up (and they can always respond afterwards).
- 4.Write up an Evaluation Manager's report if you like (this is optional but encouraged; you can get additional compensation for substantial work on a substantive report.).
- 5.We put all of this up on PubPub, get a DOI, publicize this, etc. (This process will be partially or fully automated in our updated version of PubPub.
- 6.Once the evaluations are up on PubPub, you could reach out the evaluators again with the link, in case they want to view their evaluation and the others.
- The evaluators may be allowed to revise their evaluation, e.g., if the authors find an oversight in the evaluation. (We are working on a policy for this.)
- At the moment (Nov. 2023) we don't have any explicit 'revise and resubmit' procedure, as part of the process. Authors are uncouraged to share changes they plan to make, and a (perma)-link to where their revisions can be found. They are also welcome to independently (re)-submit an updated version of their work for a later Unjournal evaluation.
Last modified 1d ago