Our initial focus is quantitative work that informs global priorities (see linked discussion), especially in economics, policy, and social science. We want to see better research leading to better outcomes in the real world (see our 'Theory of Change').

To reach these goals, we need to select "the right research" for evaluation. We want to choose papers and projects that are highly relevant, methodologically promising, and that will benefit substantially from our evaluation work. We need to optimize how we select research so that our efforts remain mission-focused and useful. We also want to make our process transparent and fair. To do this, we are building a coherent set of criteria and goals, and a specific approach to guide this process. We explore several dimensions of these criteria below.

High-level considerations for prioritizing research

When considering a piece of research to decide whether to commission it to be evaluated, we can start by looking at its general relevance as well as the value of evaluating and rating it.

1. Why is it relevant and worth engaging with?

We consider (and prioritize) the importance of the research to global priorities; its relevance to crucial decisions; the attention it is getting, the influence it is having; its direct relevance to the real world; and the potential value of the research for advancing other impactful work. We de-prioritize work that has already been credibly (publicly) evaluated. We also consider the fit of the research with our scope (social science, etc.), and the likelihood that we can commission experts to meaningfully evaluate it. As noted , some 'instrumental goals' (, , driving change, ...) also play a role in our choices.

Some features we value, that might raise the probability we consider a paper or project include the commitment and contribution to open science, the authors' engagement with our process, and the logic, communication, and transparent reasoning of the work. However, if a prominent research paper is within our scope and seems to have a strong potential for impact, we will prioritize it highly, whether or not it has these qualities.

2. Why does it need (more) evaluation, and what are some key issues and claims to vet?

We ask the people who suggest particular research, and experts in the field:

• What are (some of) the authors’ key/important claims that are worth evaluating?

• What aspects of the evidence, argumentation, methods, and interpretation, are you unsure about?

• What particular data, code, proofs, and arguments would you like to see vetted? If it has already been peer-reviewed in some way, why do you think more review is needed?

Ultimate goals: what are we trying to optimize?

Put broadly, we need to consider how this research allows us to achieve our own goals in line with our , targeting "ultimate outcomes." The research we select and evaluate should meaningfully drive positive change. One way we might see this process: “better research & more informative evaluation” → “better decision-making” → “better outcomes” for humanity and for non-human animals (i.e., the survival and flourishing of life and human civilization and values).

Prioritizing research to achieve these goals

As we weigh research to prioritize for evaluation, we need to balance directly having a positive impact against building our ability to have an impact in the future.

A. Direct impact (‘score goals now’)

Below, we adapt the (popular in effective altruism circles) to assess the direct impact of our evaluations.

Importance

What is the direct impact potential of the research?

This is a massive question many have tried to address (see sketches and links below). We respond to uncertainty around this question in several ways, including:

• Consulting a range of sources, not only EA-linked sources.

  • EA and more or less adjacent: and overviews, .

  • Non-EA, e.g., .

Neglectedness

Where is the current journal system failing GP-relevant work the most . . . in ways we can address?

Tractability

1. “Evaluability” of research: Where does the UJ approach yield the most insight or value of information?

2. Existing expertise: Where do we have field expertise on the UJ team? This will help us commission stronger evaluations.

3. "Feedback loops": Could this research influence concrete intervention choices? Does it predict near-term outcomes? If so, observing these choices and outcomes and getting feedback on the research and our evaluation can yield strong benefits.

Consideration/discussion: How much should we include research with indirect impact potential (theoretical, methodological, etc.)?

B. Sustainability: funding, support, participation

Moreover, we need to consider how the research evaluation might support the sustainability of The Unjournal and the broader general project of open evaluation. We may need to strike a balance between work informing the priorities of various audences, including:

• Relevance to stakeholders and potential supporters

• Clear connections to impact; measurability

• Support from relevant academic communities

Consideration/discussion: What will drive further interest and funding?

C. Credibility, visibility, driving positive institutional change

Finally, we consider how our choices will increase the visibility and solidify the credibility of The Unjournal and open evaluations. We consider how our work may help drive positive institutional change. We aim to:

• Interest and involve academics—and build the status of the project.

• Commission evaluations that will be visibly useful and credible.

• ‘Benchmark traditional publication outcomes’, track our predictiveness and impact.

We hope we have identified the important considerations (above); but we may be missing key points. We continue to engage discussion and seek feedback, to hone and improve our processes and approaches.

Data: what are we evaluating/considering?

We present and analyze the specifics surrounding our current evaluation data in

: An overview of what we are asking; payment and recognition details

: The Unjournal's evaluation guidelines, considering our priorities and criteria, the metrics we ask for, and how these are considered.

Other sections and subsections provide further resources, consider future initiatives, and discuss our rationales.

David Reinstein, 28 Mar 2024 I am proposing the following policies and approaches for our “Applied & Policy Stream”. We will move forward with these for now on a trial basis, but they may be adjusted. Please offer comments and ask questions in this Google doc, flagging the email 'contact@unjournal.org'

Why have an “Applied & Policy Stream”?

Much of the most impactful research is not aimed at academic audiences and may never be submitted to academic journals. It is written in formats that are very different from traditional academic outputs, and cannot be easily judged by academics using the same standards. Nonetheless, this work may use technical approaches developed in academia, making it important to gain expert feedback and evaluation.

The Unjournal can help here. However, to avoid confusion, we want to make this clearly distinct from our main agenda, which aims at impactful academically-aimed research.

This we are trialing an “Applied & Policy Stream” which will be clearly labeled as separate from our main stream. This may constitute roughly 10 or 15% of the work that we cover. Below, we refer to this as the “policy stream” for brevity.

What should be included in the Policy stream?

Our considerations for prioritizing this work are generally the same as for our academic stream – is it in the fields that we are focused on, using approaches that enable meaningful evaluation and rating? Is it already having impact (e.g., influencing grant funding in globally-important areas)? Does it have the potential for impact, and if so, is it high-quality enough that we should consider boosting its signal?

We will particularly prioritize policy and applied work that uses technical methods that need evaluation by research experts, often academics.

This could include the strongest work published on the EA Forum, as well as a range of further applied research from EA/GP/LT linked organizations such as GPI, Rethink Priorities, Open Philanthropy, FLI, HLI, Faunalytics, etc., as well as EA-adjacent organizations and relevant government white papers.

How should our (evaluation etc.) policies differ here?

Ratings/metrics: As in the academic stream, this work will be evaluated for its credibility, usefulness, communication/logic, etc. However, we are not seeking to have this work assessed by the standards of academia in a way that yields a comparison to traditional journal tiers. Evaluators: Please ignore these parts of our interface; if you are unsure if it is relevant feel free to ask.

Evaluator selection, number, pay: Generally we want to continue to select academic research experts or non-academic researchers with strong academic and methodological background to do these evaluations. A key purpose of this policy stream is largely to bring research expertise, particularly from academia, to work that is not normally scrutinized by such experts.

The compensation may be flexible as well; in some cases the work may be more involved than for the academic stream and in some cases less involved. As a starting point we will begin by offering the same compensation as for the academic stream.

Careful flagging and signposting: To preserve the reputation of our academic-stream evaluations we need to make it clear, wherever people might see this work, that it is not being evaluated by the same standards as the academic stream and doesn't “count” towards those metrics.

1. We are targeting global priorities-relevant research...

2. With the potential for impact, and with the potential for Unjournal evaluations to have an impact (see our high-level considerations and our prioritization ratings discussions).

3. Our initial focus is quantitative work that informs , especially in economics, policy, and social science, informing our .

4. We give a data presentation of the work we have already covered and the work we are prioritizing , which will be continually updated.

But what does this mean in practice? What specific research fields, topics, and approaches are we likely to classify as 'relevant to evaluate'?

We give some lists and annotated examples below.

Fields, methods, and approaches

As of January 2024 The Unjournal focuses on ...

1. Research where the fundamental question being investigated involves human behavior and beliefs and the consequences of these. This may involve markets, production processes, economic constraints, social interactions, technology, the 'market of ideas', individual psychology, government processes, and more. However, the main research question should not revolve around issues outside of human behavior, such as physical science, biology, or computer science and engineering. These areas are out of our scope (at least for now).

2. Research that is fundamentally quantitative and uses

This to generally involves the academic fields:

• Economics

• Applied Statistics (and some other applied math)

• Psychology

These discipline/field boundaries are not strict; they may adapt as we grow

Why this field/method focus?

These were chosen in light of two main factors:

1. Our founder and our team is most comfortable assessing and managing the consideration of research in these areas.

2. These fields seem to be particularly amenable to, and able to benefit from our journal-independent evaluation approach. Other fields, such as biology, are already being 'served' by strong initiatives like .

Ex.: work we included/excluded based on field/method

Outcomes, focus areas, and causes

The Unjournal's mission is to prioritize

• research with the strongest potential for a positive impact on global welfare

• where public evaluation of this research will have the greatest impact

Given this broad goal, we consider research into any cause, topic, or outcome, as long as the research involves fields, methods, and approaches within our domain (see above), and as long as the work meets our other requirements (e.g., research must be publicly shared without a paywall).

While we don't have rigid boundaries, we are nonetheless focusing on certain areas:

Fields

(As of Jan. 2024) we have mainly commissioned evaluations of work involving development economics and health-related outcomes and interventions in low-and middle-income countries.

As well as research involving

• Environmental economics, conservation, harm to human health

• The social impact of AI and emerging technologies

• Economics, welfare, and governance

We are currently prioritizing further work involving

• Psychology, behavioral science, and attitudes: the spread of misinformation; other-regarding preferences and behavior; moral circles

• Animal welfare: markets, attitudes

• Methodological work informing high-impact research (e.g., methods for impact evaluation)

We are also considering prioritizing work involving

• AI governance and safety

• Quantitative political science (voting, lobbying, attitudes)

• Political risks (including authoritarian governments and war and conflict)

Examples of work we chose to prioritize or de-prioritize based on focus area

You can request a conditional embargo by emailing us at contact@unjournal.org, or via the submission form. Please explain what sort of embargo you are asking for, and why. By default, we'd like Unjournal evaluations to be made public promptly. However, we may make exceptions in special circumstances, particularly for very early-career researchers.

If there is an early-career researcher on the authorship team, we may allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite.

For example, we might grant an embargo that lasts until after a PhD/postdoc’s upcoming job market or until after publication in a mainstream journal, with a hard maximum of 14 months. (Of course, embargoes can be ended early at the request of the authors.)

In exceptional circumstances we may consider granting a "conditional indefinite embargo."

Some examples of possible embargos (need approval)

Note: the above are all exceptions to our regular rules, examples of embargos we might or might not agree to.

Paths to suggest research

Research can be "submitted" by authors () or "suggested" by others. For a walk-through on suggesting research, see example.

There are two main paths for making suggestions: or .

Thanks for your interest in evaluating research for The Unjournal!

Who we are

The Unjournal is a nonprofit organization started in mid-2022. We commission experts to publicly evaluate and rate research. Read more about us .

We are considering asking evaluators, with compensation, to assist and engage in the process of "robustness replication." This may lead to some interesting follow-on possibilities as we build our potential collaboration with the Institute for Replication and others in this space.

We might ask evaluators discussion questions like these:

• What is the most important, interesting, or relevant substantive claim made by the authors, (particularly considering global priorities and potential interventions and responses)?

• What statistical test or evidence does this claim depend on, according to the authors?

• How confident are you in the substantive claim made?

• "Robustness checks": What specific statistical test(s) or piece(s) of evidence would make you substantially more confident in the substantive claim made?

• If a robustness replication "passed" these checks, how confident would you be then in the substantive claim? (You can also express this as a continuous function of some statistic rather than as a binary; please explain your approach.)

Background:

The Institute for Replication is planning to hire experts to do "robustness-replications" of work published in a top journal in economics and political science. Code- and data sharing is now being enforced in many or all of these journals and other important outlets. We want to support their efforts and are exploring collaboration possibilities. We are also considering how to best guide potential future robustness replication work.

Why numerical ratings?

Ultimately, we're trying to replace the question of "what tier of journal did a paper get into?" with "how highly was the paper rated?" We believe this is a more valuable metric. It can be more fine-grained. It should be less prone to gaming. It aims to reduce randomness in the process, through things like 'the availability of journal space in a particular field'. See our discussion of .

To get to this point, we need to have academia and stakeholders see our evaluations as meaningful. We want the evaluations to begin to have some value that is measurable in the way “publication in the AER” is seen to have value.

While there are some ongoing efforts towards journal-independent evaluation, these tend not use comparable metrics. Typically, they either have simple tick-boxes (like "this paper used correct statistical methods: yes/no") or they enable descriptive evaluation without an overall rating. As we are not a journal, and we don’t accept or reject research, we need another way of assigning value. We are working to determine the best way of doing this through quantitative ratings. We hope to be able to benchmark our evaluations to "traditional" publication outcomes. Thus, we think it is important to ask for both an overall quality rating and a journal ranking tier prediction.

Why these categories?

In addition to the overall assessment, we think it will be valuable to have the papers rated according to several categories. This could be particularly helpful to practitioners who may care about some concerns more than others. It also can be useful to future researchers who might want to focus on reading papers with particular strengths. It could be useful in meta-analyses, as certain characteristics of papers could be weighed more heavily. We think the use of categories might also be useful to authors and evaluators themselves. It can help them get a sense of what we think research priorities should be, and thus help them consider an overall rating.

However, these ideas have been largely ad-hoc and based on the impressions of our management team (a particular set of mainly economists and psychologists). The process is still being developed. Any feedback you have is welcome. For example, are we overemphasizing certain aspects? Are we excluding some important categories?

We are also researching other frameworks, templates, and past practice; we hope to draw from validated, theoretically grounded projects such as .

Why ask for credible intervals?

In eliciting expert judgment, it is helpful to differentiate the level of confidence in predictions and recommendations. We want to know not only what you believe, but how strongly held your beliefs are. If you are less certain in one area, we should weigh the information you provide less heavily in updating our beliefs. This may also be particularly useful for practitioners. Obviously, there are challenges to any approach. Even experts in a quantitative field may struggle to convey their own uncertainty. They may also be inherently "poorly calibrated" (see discussions and tools for ). Some people may often be "confidently wrong." They might state very narrow "credible intervals", when the truth—where measurable—routinely falls outside these boundaries. People with greater discrimination may sometimes be underconfident. One would want to consider and potentially correct for poor calibration. As a side benefit, this may be interesting for research in and of itself, particularly as The Unjournal grows. We see 'quantifying one's own uncertainty' as a good exercise for academics (and everyone) to engage in.

"Weightings" for each rating category (removed for now)

Adjustments to metrics and guidelines/previous presentations

Pre-October 2023 'ratings with weights' table, provided for reference (no longer in use)

We had included the note:

We give the previous weighting scheme in a fold below for reference, particularly for those reading evaluations done before October 2023.

As well as:

Suggested weighting: 0. Why 0?

Elsewhere in that page we had noted:

As noted above, we give suggested weights (0–5) to suggest the importance of each category rating to your overall assessment, given The Unjournal's priorities. But you don't need, and may not want to use these weightings precisely.

The weightings were presented once again along with each description in the section .

Pre-2024 ratings and uncertainty elicitation, provided for reference (no longer in use)

[FROM PREVIOUS GUIDELINES:]

You may feel comfortable giving your "90% confidence interval," or you may prefer to give a "descriptive rating" of your confidence (from "extremely confident" to "not confident").

Quantify how certain you are about this rating, either giving a 90% / interval or using our . (We prefer the 90% CI. Please don't give both.

[Previous...] Remember, we would like you to give a 90% CI or a confidence rating (1–5 dots), but not both.

And, for the 'journal tier' scale:

Previous 'descriptions of ratings intervals'

[Previous guidelines]: The description folded below focuses on the "Overall Assessment." Please try to use a similar scale when evaluating the category metrics.

See also

This page explains the value of the metrics we are seeking from evaluators.

You can download a pdf version of these guidelines here (updated March 2024).

What we'd like you to do

1. Write an evaluation of the target paper or project, similar to a standard, high-quality referee report. Please identify the paper's main claims and carefully assess their validity, leveraging your own background and expertise.

Writing the evaluation (aka 'the review')

In writing your evaluation and providing ratings, please consider the following.

The Unjournal's expectations and criteria

In many ways, the written part of the evaluation should be similar to a report an academic would write for a traditional high-prestige journal (e.g., see some 'conventional guidelines' ). Most fundamentally, we want you to use your expertise to critically assess the main claims made by the authors. Are the claims well-supported? Are the assumptions believable? Are the methods are appropriate and well-executed? Explain why or why not.

However, we'd also like you to pay some consideration to our priorities:

1. Advancing our knowledge and practice

2. Justification, reasonableness, validity, and robustness of methods

3. Logic and communication

4. Open, communicative, replicable science

See our for more details on each of these. Please don't structure your review according to these metrics, just pay some attention to them.

If you have questions about the authors’ work, you can ask them anonymously: we will facilitate this.

We want you to evaluate the most recent/relevant version of the paper/project that you can access. If you see a more recent version than the one we shared with you, please let us know.

Target audiences

We designed this process to balance three considerations with three target audiences. Please consider each of these:

1. Crafting evaluations and ratings that help researchers and policymakers judge when and how to rely on this research. For Research Users.

2. Ensuring these evaluations of the papers are comparable to current journal tier metrics, to enable them to be used to determine career advancement and research funding. For Departments, Research Managers, and Funders.

3. Providing constructive feedback to Authors

We discuss this, and how it relates to our impact and "theory of change", .

Quantitative metrics

We ask for a set of nine quantitative metrics. For each metric, we ask for a score and a 90% credible interval. We describe these in detail below. (We explain .)

Percentile rankings

For some questions, we ask for a percentile ranking from 0-100%. This represents "what proportion of papers in the reference group are worse than this paper, by this criterion". A score of 100% means this is essentially the best paper in the reference group. 0% is the worst paper. A score of 50% means this is the median paper; i.e., half of all papers in the reference group do this better, and half do this worse, and so on.

Here* the population of papers should be all serious research in the same area that you have encountered in the last three years.

Midpoint rating and credible intervals

For each metric, we ask you to provide a 'midpoint rating' and a 90% credible interval as a measure of your uncertainty. Our interface provides slider bars to express your chosen intervals:

The table below summarizes the percentile rankings.

Overall assessment

Percentile ranking (0-100%)

Judge the quality of the research heuristically. Consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.

Claims, strength and characterization of evidence **

Do the authors do a good job of (i) stating their main questions and claims, (ii) providing strong evidence and powerful approaches to inform these, and (iii) correctly characterizing the nature of their evidence?

Methods: Justification, reasonableness, validity, robustness

Percentile ranking (0-100%)

Are the methods used well-justified and explained; are they a reasonable approach to answering the question(s) in this context? Are the underlying assumptions reasonable?

Are the results and methods likely to be robust to reasonable changes in the underlying assumptions? Does the author demonstrate this?

Avoiding bias and (QRP): Did the authors take steps to reduce bias from opportunistic reporting and QRP? For example, did they do a strong pre-registration and pre-analysis plan, incorporate multiple hypothesis testing corrections, and report flexible specifications?

Advancing our knowledge and practice

Percentile ranking (0-100%)

To what extent does the project contribute to the field or to practice, particularly in ways that are relevant to global priorities and impactful interventions?

(Applied stream: please focus on ‘improvements that are actually helpful’.)

Do the paper's insights inform our beliefs about important parameters and about the effectiveness of interventions?

Does the project add useful value to other impactful research?

We don't require surprising results; sound and well-presented null results can also be valuable.

Logic and communication

Percentile ranking (0-100%)

Are the goals and questions of the paper clearly expressed? Are concepts clearly defined and referenced?

Is the reasoning "transparent"? Are assumptions made explicit? Are all logical steps clear and correct? Does the writing make the argument easy to follow?

Are the conclusions consistent with the evidence (or formal proofs) presented? Do the authors accurately state the nature of their evidence, and the extent it supports their main claims?

Are the data and/or analysis presented relevant to the arguments made? Are the tables, graphs, and diagrams easy to understand in the context of the narrative (e.g., no major errors in labeling)?

Open, collaborative, replicable research

Percentile ranking (0-100%)

This covers several considerations:

Replicability, reproducibility, data integrity

Would another researcher be able to perform the same analysis and get the same results? Are the methods explained clearly and in enough detail to enable easy and credible replication? For example, are all analyses and statistical tests explained, and is code provided?

Is the source of the data clear?

Is the data made as available as is reasonably possible? If so, is it clearly labeled and explained??

Consistency

Do the numbers in the paper and/or code output make sense? Are they internally consistent throughout the paper?

Useful building blocks

Do the authors provide tools, resources, data, and outputs that might enable or enhance future work and meta-analysis?

Relevance to global priorities, usefulness for practitioners**

Are the paper’s chosen topic and approach likely to be useful to

Does the paper consider real-world relevance and deal with policy and implementation questions? Are the setup, assumptions, and focus realistic?

Do the authors report results that are relevant to practitioners? Do they provide useful quantified estimates (costs, benefits, etc.) enabling practical impact quantification and prioritization?

Do they communicate (at least in the abstract or introduction) in ways policymakers and decision-makers can understand, without misleading or oversimplifying?

Journal ranking tiers

To help universities and policymakers make sense of our evaluations, we want to benchmark them against how research is currently judged. So, we would like you to assess the paper in terms of journal rankings. We ask for two assessments:

1. a normative judgment about 'how well the research should publish';

2. a prediction about where the research will be published.

Journal ranking tiers are on a 0-5 scale, as follows:

• 0/5: "Won't publish/little to no value". Unlikely to be cited by credible researchers

• 1/5: OK/Somewhat valuable journal

• 2/5: Marginal B-journal/Decent field journal

We encourage you to consider a non-integer score, e.g. 4.6 or 2.2.

As before, we ask for a 90% credible interval.

What journal ranking tier should this work be published in?

Journal ranking tier (0.0-5.0)

Assess this paper on the journal ranking scale described above, considering only its merit, giving some weight to the category metrics we discussed above.

Equivalently, where would this paper be published if:

1. the journal process was fair, unbiased, and free of noise, and that status, social connections, and lobbying to get the paper published didn’t matter;

2. journals assessed research according to the category metrics we discussed above.

What journal ranking tier will this work be published in?

Journal ranking tier (0.0-5.0)

The midpoint and 'credible intervals': expressing uncertainty

What are we looking for and why?

We want policymakers, researchers, funders, and managers to be able to use The Unjournal's evaluations to update their beliefs and make better decisions. To do this well, they need to weigh multiple evaluations against each other and other sources of information. Evaluators may feel confident about their rating for one category, but less confident in another area. How much weight should readers give to each? In this context, it is useful to quantify the uncertainty.

But it's hard to quantify statements like "very certain" or "somewhat uncertain" – different people may use the same phrases to mean different things. That's why we're asking for you a more precise measure, your credible intervals. These metrics are particularly useful for meta-science and meta-analysis.

You are asked to give a 'midpoint' and a 90% credible interval. Consider this as the smallest interval that you believe is 90% likely to contain the true value. See the fold below for further guidance.

Claim identification, assessment, and implications

We are now asking evaluators for “claim identification and assessment” where relevant. This is meant to help practitioners use this research to inform their funding, policymaking, and other decisions. It is not intended as a metric to judge the research quality per se. This is not required but we will reward this work.

.

Survey questions

Lastly, we ask evaluators about their background, and for feedback about the process.

Other guidelines and notes

Length/time spent: This is up to you. We welcome detail, elaboration, and technical discussion.

Our data protection statement is linked .

How to write a good review (general conventional guidelines)

The Unjournal Evaluators have the option of remaining anonymous (see ). Where evaluators choose this, we will carefully protect this anonymity, aiming at a high standard of protection, as good as or better than traditional journals. We will give evaluators the option to take extra steps to safeguard this further. We are offering anonymity in perpetuity to those who request it. (As well as anonymity on other terms to those who request it, on explicitly mutually agreed upon terms.)

If they choose to stay anonymous, there should be no way for authors to be able to ‘guess’ who has reviewed their work.

Some key principles/rules

It's a norm in academia that people do reviewing work for free. So why is The Unjournal paying evaluators?

From a recent survey of economists:

We estimate that the average (median) respondent spends 12 (9) working days per year on refereeing. The top 10% of the distribution dedicates 25 working days or more, which is quite substantial considering refereeing is usually unpaid.

General reasons to pay reviewers

Economics, turnaround times

The peer-review process in economics is widely-argued to be too slow and lengthy. But there is evidence that payments may help improve this.

, they note that few economics journals currently pay reviewers (and these payments tend to be small (e.g., JPE and AER paid $100 at the time). However, they also note, citing several papers:

The existing evidence summarized in Table 5 suggests that offering financial incentives could be an effective way of reducing turnaround time.

Equity and inclusivity

notes that the work of reviewing is not distributed equally. To the extent that accepting to do a report is based on individual goodwill, the unpaid volunteer model could be seen to unfairly penalize more generous and sympathetic academics. Writing a certain number of referee reports per year is generally considered part of "academic service". Academics put this on their CVs, and it may lead to being on the board of a journal which is valued to an extent. However, this is much less attractive for researchers who are not tenured university professors. Paying for this work would do a better job of including them in the process.

Incentivizing useful, unbiased evaluations

'Payment for good evaluation work' may also lead to fair and more useful evaluations.

In the current system academics may take on this work in large part to try to impress journal editors and get favorable treatment from them when they submit their own work. They may also write reviews in particular ways to impress these editors.

For less high-prestige journals, to get reviewers, editors often need to lean on their personal networks, including those they have power relationships with.

Reviewers are also known to strategically try to get authors to cite and praise the reviewer's own work. They maybe especially critical to authors they see as rivals.

To the extent that reviewers are doing this as a service they are being paid for, these other motivations will be comparatively somewhat less important. The incentives will be more in line with doing evaluations that are seen as valuable by the managers of the process, in order to get chosen for further paid work. (And, if evaluations are public, the managers can consider the public feedback on these reports as well.)

Reasons for The Unjournal to pay evaluators

1. We are not ‘just another journal.’ We need to give incentives for people to put effort into a new system and help us break out of the old inferior equilibrium.

2. In some senses, we are asking for more than a typical journal. In particular, our evaluations will be made public and thus need to be better communicated.

3. We cannot rely on 'reviewers taking on work to get better treatment from editors in the future.' This does not apply to our model, as we don't have editors make any sort of ‘final accept/reject decision’

As noted in Process: prioritizing research, we ask people who suggest research to provide a numerical 0-100 rating:

We also ask people within our team to act as 'assessors' to give as second and third opinions on this. This 'prioritization rating' is one of the criteria we will use to determine whether to commission research to be evaluated (along with author engagement, publication status, our capacity and expertise, etc.) Again, see the previous page for the current process.

So what goes into this "prioritization rating"; what does it mean?

We are working on a set of notes on this, fleshing this out and giving specific examples. At the moment this is available to members of our team only (ask for access to "Guidelines for prioritization ratings (internal)"). We aim to share a version of this publicly once it converges, and once we can get rid of arbitrary sensitive examples.

Some key points

I. This is not the evaluation itself. It is not an evaluation of the paper's merit per se:

• Influential work, and prestigious work in influential areas may be highly prioritized regardless of its rigor and quality

• The prioritization rating might consider quality for work that seems potentially impactful, which does not seem particularly prestigious or influential. Here aspects like writing clarity, methodological rigor, etc., might put it 'over the bar'. However, even here these will tend to be based on rapid and shallow assessments, and should not be seen as meaningful evaluations of research merit.

II. These ratings will be considered along with the discussion by the field team and the management. Thus is helpful if you give a justification and explanation for your stated rating.

One possible way of considering the rating criteria

Key attributes/factors

Define/consider the following ‘attributes’ of a piece of research:

1. Global decision-relevance/VOI: Is this research decision-relevant to high-value choices and considerations that are important for global priorities and global welfare?

2. Prestige/prominence: Is the research already prominent/valued (esp. in academia), highly cited, reported on, etc?

3. Influence: Is the work already influencing important real-world decisions and considerations?

Obviously, these are not binary factors; there is a continuum for each. But for the sake of illustration, consider the following flowcharts.

Prestigious work

"Fully baked": Sometimes prominent researchers release work (e.g., on NBER) that is not particularly rigorous or involved, which may have been put together quickly. This might be research that links to a conference they are presenting at, to their teaching, or to specific funding or consulting. It may be survey/summary work, perhaps meant for less technical audiences. The Unjournal tends not to prioritize such work, or at least not consider it in the same "prestigious" basket (although there will be exceptions). In the flowchart above, we contrast this with their "fully-baked" work.

Decision-relevant, prestigious work: Suppose the research is both ‘globally decision-relevant’ and prominent. Here, if the research is in our domain, we probably want to have it publicly evaluated. This is basically the case regardless of its apparent methodological strength. This is particularly true if it has been recently made public (as a working paper), if it has not yet been published in a highly-respected peer-reviewed journal, and if there are non-straightforward methodological issues involved.

Prestigious work that seems less globally-relevant: We generally will not prioritize this work unless it adds to our mission in other ways (see, e.g., our ‘sustainability’ and ‘credibility’ goals ). In particular we will prioritize such research more if:

• It is presented in innovative, transparent formats (e.g., dynamic documents/open notebooks, sharing code and data)

• The research indirectly supports more globally-relevant research, e.g., through…

  • Providing methodological tools that are relevant to that ‘higher-value’ work

Less prestigious work

(If the flowchart below does not render, please refresh your browser; you may have to refresh twice.)

Decision-relevant, influential (but less prestigious) work: E.g., suppose this research might be cited by a major philanthropic organization as guiding its decision-making, but the researchers may not have strong academic credentials or a track record. Again, if this research is in our domain, we probably want to have it publicly evaluated. However, depending on the rigor of the work and the way it is written, we may want to explicitly class this in our ‘non-academic/policy’ stream.

Decision-relevant, less prestigious, less-influential work: What about for less-prominent work with fewer academic accolades that is not yet having an influence, but nonetheless seems to be globally decision-relevant? Here, our evaluations seem less likely to have an influence unless the work seems potentially strong, implying that our evaluations, rating, and feedback could boost potentially valuable neglected work. Here, our prioritization rating might focus more on our initial impressions of things like …

• Methodological strength (this is a big one!)

• Rigorous logic and communication

• Open science and robust approaches

Again: the prioritization process is not meant to be an evaluation of the work in itself. It’s OK to do this in a fairly shallow way.

In future, we may want to put together a loose set of methodological ‘suggestive guidelines’ for work in different fields and areas, without being too rigid or prescriptive. (To do: we can draw from some existing frameworks for this [ref].)

Curating and publishing evaluations, linked to research

• Unjournal PubPub page

  • Previous/less emphasized: Society

• Evaluations and author response are given DOI's, enter the bibliometric record

  • Future consideration:

    • "publication tier" of authors' responses as a workaround to encode aggregated evaluation
• Sharing evaluation data on public Github repo (see )

Aggregating evaluators' ratings and predictions

We aim to elicit the experiment judgment from Unjournal evaluators efficiently and precisely. We aim to communicate this quantitative information concisely and usefully, in ways that will inform policymakers, philanthropists, and future researchers.

In the short run (in our pilot phase), we will attempt to present simple but reasonable aggregations, such as simple averages of midpoints and confidence-interval bounds. However, going forward, we are consulting and incorporating the burgeoning academic literature on "aggregating expert opinion." (See, e.g., ; ; ; .)

We are working on this in our public data presentation (Quarto notebook) .

Other communication

We are considering...

• Syntheses of evaluations and author feedback

• Input to prediction markets, replication projects, etc.

• Less technical summaries and policy-relevant summaries, e.g., for the , , or mainstream long-form outlets

Submission/evaluation funnel

As we are paying evaluators and have limited funding, we cannot evaluate every paper and project. Papers enter our database through

1. submission by authors;

1. Project submission, selection and prioritization

People and organizations submit their own research or suggest research they believe may be high-impact. The Unjournal also directly monitors key sources of research and research agendas. Our team then systematically prioritizes this research for evaluation. See the link below for further details.

2. Evaluation

1. We choose an evaluation manager for each research paper or project. They commission and compensate expert evaluators to rate and discuss the research, following our evaluation template and . The original research authors are given a chance to publicly respond before we post these evaluations. See the link below for further details.

3. Communicating results

We make all of this evaluation work public on , along with an evaluation summary. We create DOIs for each element and submit this work to scholarly search engines. We also present a summary and analysis of our .

We outline some further details in the link below.

Flowchart

See the link below for a full 'flowchart' map of our evaluation workflow

15 Dec 2023: Our main current process involves

• Submitted and (internally/externally) suggested research

• Prioritization ratings and discussion by Unjournal field specialists

• Feedback from field specialist area teams

• A final decision by the management team, guided by the above

See (also embedded below) for more details of the proposed process.

Second track: Direct evaluation of prominent work

In addition to soliciting research submissions by authors, we directly prioritize unsubmitted research for evaluation, with a specific process and set of rules, outlined below.

1. Choose a set of "top-tier working paper series" and medium-to-top-tier journals.

   This program started with the . We expanded this beyond NBER to research posted in other exclusive working paper archives and to work where all authors seem to be prominent, secure, and established. See .

2. Identify relevant papers in this series, following our stated criteria (i.e., , strength, ). For NBER this tends to include

   • recently released work in the early stages of the journal peer-review process, particularly if it addresses a timely subject; as well as

   • work that has been around for many years, is widely cited and influential, yet has never been published in a peer-reviewed journal.

We do this systematically and transparently; authors shouldn't feel singled out nor left out.

1. Notify the work's authors that The Unjournal plans to commission evaluations. We're not asking for permission, but

   • making them aware of The Unjournal, the process, the , and the authors' opportunities to engage with the evaluation and publicly respond to the evaluation before it is made public;

   • letting us know if we have the most recent version of the paper, and if updates are coming soon;

Direct evaluation: eligibility rules and guidelines

NBER

All NBER working papers are generally eligible, but watch for exceptions where authors seem vulnerable in their career. (And remember, we contact authors, so they can plead their case.)

CEPR

We treat these on a case-by-case basis and use discretion. All CEPR members are reasonably secure and successful, but their co-authors might not be, especially if these co-authors are PhD students they are supervising.

Journal-published papers (i.e., 'post-publication evaluation')

In some areas and fields (e.g., psychology, animal product markets) the publication process is relatively rapid or it may fail to engage general expertise. In general, all papers that are already published in peer-reviewed journals are eligible for our direct track.

Papers or projects posted in any other working paper (pre-print) series

These are eligible (without author permission) if all authors

• have tenured or ‘long term’ positions at well-known, respected universities or other research institutions, or

• have tenure-track positions at top universities (e.g., top-20 globally by some credible rankings), or

• are clearly not pursuing an academic career (e.g., the "partner at the aid agency running the trial").

On the other hand, if one or more authors is a PhD student close to graduation or an untenured academic outside a "top global program,’’ then we will ask for permission and potentially offer an embargo.

• A possible exception to this exception: If the PhD student or untenured academic is otherwise clearly extremely high-performing by conventional metrics; e.g., an REStud "tourist" or someone with multiple published papers in top-5 journals. In such cases the paper might be considered eligible for direct evaluation.

The flowchart below focuses on the evaluation part of our process.

Describing key steps in the flowchart

(Section updated 1 August 2023)

1. Submission/selection (multiple routes)

   1. Author (A) submits work (W), creates new submission (submits a URL and DOI), through our platform or informally.

      • Author (or someone on their behalf) can complete a submission form; this includes a potential "request for embargo" or other special treatment.

   2. Managers and field specialists select work (or the project is submitted independently of authors) and the management team agrees to prioritize it.

2. Prioritization

   • Following author submission ...

     • Manager(s) (M) and Field Specialists (FS) prioritize work for review (see ).
3. M assigns an Evaluation Manager (EM – typically part of our ) to selected project.

4. EM invites evaluators (aka "reviewers") and shares the paper to be evaluated along with (optionally) a brief summary of why The Unjournal thinks it's relevant, and what we are asking.

   • Potential evaluators are given full access to (almost) all information submitted by the author and M, and notified of any embargo or special treatment granted.

   • EM may make special requests to the evaluator as part of a management policy (e.g., "signed/unsigned evaluation only," short deadlines, extra incentives as part of an agreed policy, etc.).

5. Evaluator accepts or declines the invitation to review, and if the former, agrees on a deadline (or asks for an extension).

   • If the evaluator accepts, the EM shares full guidelines/evaluation template and specific suggestions with the evaluator.

6. Evaluator completes an evaluation form.

7. Evaluator submits evaluation including numeric ratings and predictions, plus "CI's" for these.

   • Possible addition (future plan): Reviewer asks for minor revisions and corrections; see "How revisions might be folded in..." in the fold below.

8. EM collates all evaluations/reviews, shares these with Author(s).

   • Evaluator must be very careful not to share evaluators' identities at this point.

     • This includes caution to avoid accidentally-identifying information, especially where

9. Author(s) read(s) evaluations, given two working weeks to submit responses.

   • If there is an embargo, there is more time to do this, of course.

10. EM creates evaluation summary and "EM comments."

11. EM or UJ team publishes each element on our space as a separate "pub" with a DOI for each (unless embargoed):

    1. Summary and EM comments

       • With a prominent section for the "ratings data tables"
12. Authors and evaluators are informed once elements are on PubPub; next steps include promotion, checking bibliometrics, etc.

13. ("Ratings and predictions data" to enter an additional public database.)

Note that we intend to automate and integrate many of the process into an editorial-management-like system in PubPub.

Consideration for the future: enabling "minor revisions"

In our current (8 Feb 2023 pilot) phase, we have the evaluators consider the paper "as is," frozen at a certain date, with no room for revisions. The authors can, of course, revise the paper on their own and even pursue an updated Unjournal review; we would like to include links to the "permanently updated version" in the Unjournal evaluation space.

After the pilot, we may consider making minor revisions part of the evaluation process. This may add substantial value to the papers and process, especially where evaluators identify straightforward and easily-implementable improvements.

Why would we (potentially) consider only minor revisions?

We don't want to replicate the slow and inefficient processes of the traditional system. Essentially, we want evaluators to give a report and rating as the paper stands.

We also want to encourage papers as projects. The authors can improve it, if they like, and resubmit it for a new evaluation.

Details of submissions to The Unjournal

Note: This section largely repeats content in our guide for researchers/authors, especially our FAQ on "why engage."

Jan. 2024: We have lightly updated this page to reflect our current systems.

What we are looking for

We describe the nature of the work we are looking to evaluate, along with examples, in . Update 2024: This is now better characterized under and .

If you are interested in submitting your work for public evaluation, we are looking for research which is relevant to global priorities—especially quantitative social sciences—and impact evaluations. Work that would benefit from further feedback and evaluation is also of interest.

Your work will be evaluated using our evaluation guidelines and metrics. You can read these before submitting.

Important Note: We are not a journal. By having your work evaluated, you will not be giving up the opportunity to have your work published in a journal. We simply operate a system that allows you to have your work independently evaluated.

If you think your work fits our criteria and would like it to be publicly evaluated, please submit your work through .

If you would like to submit more than one of your papers, you will need to complete a new form for each paper you submit.

Conditional embargo on the publishing of evaluations

By default, we would like Unjournal evaluations to be made public. We think public evaluations are generally good for authors, as explained . However, in special circumstances and particularly for very early-career researchers, we may make exceptions.

If there is an early-career researcher on the author team, we will allow authors to "embargo" the publication of the evaluation until a later date. This date is contingent, but not indefinite. The embargo lasts until after a PhD/postdoc’s upcoming job search or until it has been published in a mainstream journal, unless:

• the author(s) give(s) earlier permission for release; or

• until a fixed upper limit of 14 months is reached.

If you would like to request an exception to a public evaluation, you will have the opportunity to explain your reasoning in the submission form.

See "" for more detail, and examples.

Why might an author want to engage with The Unjournal?

1. The Unjournal presents an additional opportunity for evaluation of your work with an emphasis on impact.

2. Substantive feedback will help you improve your work—especially useful for young scholars.

3. Ratings can be seen as markers of credibility for your work that could help your career advancement at least at the margin, and hopefully help a great deal in the future. You also gain the opportunity to

What we might ask of authors

If we consider your work for public evaluation, we may ask for some of the items below, although most are optional. We will aim to make this a very light touch for authors.

1. A link to a non-paywalled, hosted version of your work (in any format—PDFs are not necessary) that can be given a Digital Object Identifier (DOI). Again, we will not be "publishing" this work, just evaluating it.

2. A link to data and code, if possible. We will work to help you to make it accessible.

3. Assignment of two evaluators who will be paid to assess your work. We will likely keep their identities confidential, although this is flexible depending on the reviewer. Where it seems particularly helpful, we will facilitate a confidential channel to enable a dialogue with the authors. One person on our managing team will handle this process.

• By completing the submission form, you are providing your permission for us to post the evaluations publicly unless you request an embargo.

• You will have a two-week window to respond through our platform before anything is posted publicly. Your responses can also be posted publicly.

For more information on why authors may want to engage and what we may ask authors to do, please see .