Project submission, selection and prioritization
Last updated
Was this helpful?
Last updated
Was this helpful?
As we are paying evaluators and have limited funding, we cannot evaluate every paper and project. Papers enter our database through:
submission by authors;
our own searches (e.g., searching syllabi, forums, working paper archives, and white papers); and
s from other researchers, practitioners, and members of the public, and recommendations from . We have posted more detailed instructions for .
Our management team rates the suitability of each paper according to the criteria discussed below and .
We have followed a few procedures for finding and prioritizing papers and projects. In all cases, we require more than one member of our research-involved team (field specialist, managers, etc.) to support a paper before prioritizing it.
We are building a grounded systematic procedure with criteria and benchmarks. We also aim to give managers and field specialists some autonomy in prioritizing key papers and projects. As noted elsewhere, we are considering targets for particular research areas and sources.
See our basic process (as of Dec. 2023) for prioritizing work: Process: prioritizing research
Through October 2022: For the papers or projects at the top of our list, we contacted the authors and asked if they wanted to engage, only pursuing evaluation if agreed.
In our , we inform authors but do not request permission. For this track, we have largely focused on working papers.
In deciding which papers or projects to send out to paid evaluators, we have considered the following issues. for each paper or project to evaluators before they write their evaluations.
What are (some of) the authors’ main claims that are worth carefully evaluating? What aspects of the evidence, argumentation, methods, interpretation, etc., is the team unsure about? What particular data, code, proof, etc., would they like to see vetted? If it has already been peer-reviewed in some way, why do they think more review is needed?
How well has the author engaged with the process? Do they need particular convincing? Do they need help making their engagement with The Unjournal successful?
See What research to target? for further discussion of prioritization, scope, and strategic and sustainability concerns.
Consider: , field relevance, open science, authors’ engagement, data and reasoning transparency. In gauging this relevance, the team may consider the , but not too rigidly.