I (David Reinstein) am an economist who left UK academia after 15 years to pursue a range of projects (see my web page). One of these is The Unjournal:

The Unjournal (with funding from the Long Term Future Fund and the Survival and Flourishing Fund) organizes and funds public-journal-independent feedback and evaluation, paying reviewers for their work. We focus on research that is highly relevant to global priorities, especially in economics, social science, and impact evaluation. We encourage better research by making it easier for researchers to get feedback and credible ratings on their work.

Evaluators

We want researchers who are interested in doing evaluation work for The Unjournal. We pay an average of $400-$500 per complete and on-time evaluation, and we award monetary prizes for the strongest work. Right now we are particularly looking for economists and people with quantitative and policy-evaluation skills. We describe what we are asking evaluators to do here: essentially a regular peer review with some different emphases, plus providing a set of quantitative ratings and predictions. Your evaluation content would be made public (and receive a DOI, etc.), but you can choose if you want to remain anonymous or not.

To sign up to be part of the pool of evaluators or to get involved in The Unjournal project in other ways, please fill out this brief form or email contact@unjournal.org.

Research

We welcome suggestions for particularly impactful research that would benefit from (further) public evaluation. We choose research for public evaluation based on an initial assessment of methodological strength, openness, clarity, relevance to global priorities, and the usefulness of further evaluation and public discussion. We sketch these criteria here, and discuss some potential examples here (see research we have chosen and evaluated at unjournal.pubpub.org, and a larger list of research we're considering here).

If you have research—your own or others—that you would like us to assess, please fill out this form. You can submit your own work here (or by contacting ). Authors of evaluated papers will be eligible for our Impactful Research Prizes (details).

Feedback

We are looking for both feedback on and involvement in The Unjournal project. Feel free to reach out at .

View our data protection statement

Some key sections and subsections

Learn more about The Unjournal, our goals and policies

For authors, evaluators, etc.

Writeups of the main points for a few different audiences

Important benefits of journal-independent public evaluation and The Unjournal's approach, with links to deeper commentary

How we choose papers/projects to evaluate, how we assign evaluators, and so on

Other resources and reading

Groups we work with; comparing approaches

What research are we talking about? What will we cover?

Detail, progress, and internal planning

These are of more interest to people within our team; we are sharing these in the spirit of transparency.

A "best feasible plan" for going forward

Successful proposals (ACX, SFF), other applications, initiatives

Note: we have moved some of this "internal interest content" over to our Coda.io knowledge base.

To indicate your potential interest in roles at The Unjournal, such as those described below, please fill out this quick survey form and link (or upload) your CV or webpage.

• If you already filled out this form for a role that has changed titles, don’t worry. You will still be considered for relevant and related roles in the future.

• If you add your name to this form, we may contact you to offer you the opportunity to do paid project work and paid work tasks.

Furthermore, if you are interested in conducting paid research evaluation for The Unjournal, or in joining our advisory board, please complete the form linked here.

Feel free to contact contact@unjournal.org with any questions.

Quick links to role descriptions below

Administration, operations and management roles

Research & operations-linked roles & projects

Standalone project: Impactful Research Scoping (temp. pause)

Additional information

Express interest in any of these roles in our survey form.

The Unjournal, a not-for-profit collective under the umbrella and fiscal sponsorship of the Open Collective Foundation, is an equal-opportunity employer and contractor. We are committed to creating an inclusive environment for all employees, volunteers, and contractors. We do not discriminate on the basis of race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetic information, disability, age, or veteran status.

See our data protection statement linked here.

In a nutshell

The Unjournal seeks to make rigorous research more impactful and impactful research more rigorous. We are a team of researchers, practitioners, and open science advocates led by David Reinstein.

The Unjournal encourages better research by making it easier for researchers to get feedback and credible ratings. We coordinate and fund public journal-independent expert evaluation of hosted papers and dynamically presented projects. We publish evaluations, ratings, manager summaries, author responses, and links to evaluated research on our PubPub page.

As the name suggests, we are not a journal!

We work independently of traditional academic journals. We're building an open platform and a sustainable system for feedback, ratings, and assessment. We're currently focusing on quantitative work that informs global priorities in economics, social science, and policy.

How to get involved?

We're looking for research to evaluate, as well as evaluators. You can submit your own research here, or suggest research using this form. We offer financial prizes for suggesting research we end up evaluating. If you want to be an evaluator, apply here. You can use the same form to express your interest in joining our management team, advisory board, or reviewer pool. For more information, see our how to get involved guide.

Why The Unjournal? Peer review is great, but conventional academic publication processes are wasteful, slow, and rent-extracting. They discourage innovation and prompt researchers to focus more on "gaming the system" than on the quality of their research. We will provide an immediate alternative, and at the same time, offer a bridge to a more efficient, informative, useful, and transparent research evaluation system.

Does The Unjournal charge any fees?

No. We're a US-registered tax-exempt 501(c)(3) nonprofit, and we don't charge fees for anything. We compensate evaluators for their time and we even award prizes for strong research and evaluation work, in contrast to most traditional journals. We do so thanks to funding from the Long-Term Future Fund and Survival and Flourishing Fund.

At some point in the future, we might consider sliding-scale fees for people or organizations submitting their work for Unjournal evaluation, or for other services. If we do this, it would simply be a way to cover the compensation we pay evaluators and to cover our actual costs. Again, we are a nonprofit and we will stay that way.

How do we do this?

1. Research submission/identification and selection: We identify, solicit, and select relevant research work to be hosted on any open platform in any format that can gain a time-stamped DOI. Authors are encouraged to present their work in the ways they find most comprehensive and understandable. We support the use of dynamic documents and other formats that foster replicability and open science. (See: the benefits of dynamic docs).

2. Paid evaluators (AKA "reviewers"): We compensate evaluators (essentially, reviewers) for providing thorough feedback on this work. (Read more: Why do we pay?)

3. Eliciting quantifiable and comparable metrics: We aim to establish and generate credible measures of research quality and usefulness. We benchmark these against traditional previous measures (such as journal tiers) and assess the reliability, consistency, and predictive power of these measures. (Read more: Why quantitative metrics?)

4. Public evaluation: We publish the evaluation packages (including reports, ratings, author responses, and manager summaries) on our PubPub community. Making evaluation public facilitates dialogue, and supports transparency, impact, understanding, and research progress.

5. Linking, not publishing: Our process is not "exclusive." Authors can submit their work to a journal (or other evaluation service) at any time. This approach also allows us to benchmark our evaluations against traditional publication outcomes.

6. Prizes: We award financial prizes and hold public events to recognize the most credible, impactful, useful, and insightful research, as well as strong engagement with our evaluation process.

7. Transparency: We aim for maximum transparency in our processes and judgments.

Change is hard: overcoming academic inertia

Academics and funders have complained about this stuff for years and continue to do so every day on social media . . . and we're fairly confident our critiques of the traditional review and publication process will resonate with most readers.

So why haven't academia and the research community been able to move to something new? There is a difficult collective action problem. Individual researchers and universities find it risky to move unilaterally. But we believe we have a good chance of finally changing this model and moving to a better equilibrium. How? We will:

• Take risks: Many members of The Unjournal management are not traditional academics; we can stick our necks out. We are also recruiting established senior academics who are less professionally vulnerable.

• Bring in new interests, external funding, and incentives: There are a range of well-funded and powerful organizations—such as the Sloan Foundation and Open Philanthropy—with a strong inherent interest in high-impact research being reliable, robust, and reasoning-transparent. This support can fundamentally shift existing incentive structures.

• Allow less risky "bridging steps": As noted above, The Unjournal allows researchers to submit their work to traditional journals. In fact, this will provide a benchmark to help build our quantitative ratings and demonstrate their value.

• Communicate with researchers and stakeholders to make our processes easy, clear, and useful to them.

• Make our output useful, in the meantime: It may take years for university departments and grant funders to incorporate journal-independent evaluations as part of their metrics and reward systems. The Unjournal can be somewhat patient: our evaluation, rating, feedback, and communication are already providing a valuable service to authors, policymakers, and other researchers.

• Leverage new technology: A new set of open-access and AI-powered tools makes what we are trying to do easier, and makes formats other than static PDFs more useful every day.

• Reward early adopters with prizes and recognition: We can replace "fear of standing out" with "fear of missing out." In particular, authors and research institutions that commit to publicly engaging with evaluations and critiques of their work should be commended and rewarded. And we are doing this.

Our objectives

This GitBook is a knowledge base that supplements our main public page, unjournal.org. It serves as a platform to organize our ideas and resources and track our progress towards our dual objectives:

1. Making "peer evaluation and rating" of open projects into a standard high-status outcome in academia and research, specifically within economics and social sciences. This stands in contrast to the conventional binary choice of accepting or rejecting papers to be published as PDFs and other static formats.

2. Building a cohesive and efficient system for publishing, accruing credibility, and eliciting feedback for research aligned with effective altruism and global priorities. Our ultimate aim is to make rigorous research more impactful, and impactful research more rigorous.

Where do I find...? Where do I go next?

See Content overview

In brief

(see our #in-a-nutshell) wants your involvement, help, and feedback. We offer rewards and strive to compensate people for their time and effort.

1. Join our team: Complete this form (about 3–5 min) to apply for our...

   1. Evaluator pool: to be eligible to be commissioned and paid to evaluate and rate research, mainly in quantitative social science and policy

   2. Field specialist teams: help identify, prioritize, and manage research evaluation in a particular field or cause area

   3. Management team or advisory board, to be part of our decision-making

2. Suggest research for us to assess using this form. We offer bounty rewards. Submit your own research here, or by contacting

3. Do an Independent Evaluation to build your portfolio, receive guidance, and be eligible for promotion and prizes. See details Independent evaluations (trial)

4. Suggest "Pivotal questions" for us to focus on

Give us feedback: Is anything unclear? What could be improved? Email contact@unjournal.org. We will offer rewards for the most useful suggestions.

Overview and call

David Reinstein is the founder and co-director of The Unjournal. The organization is currently looking for field specialists and evaluators, as well as suggestions for relevant work for The Unjournal to evaluate.

The Unjournal team is building a system for credible, public, journal-independent feedback and evaluation of research.

We maintain an open call for participants for four different roles:

1. Management Committee members (involving honorariums for time spent)

2. Advisory Board members (no time commitment)

3. Field Specialists (who will often also be on the Advisory Board)

4. A pool of Evaluators (who will be paid for their time and their work; we also draw evaluators from outside this pool)

The roles are explained in more detail here. You can express your interest (and enter our database) here.

Evaluators

We will reach out to evaluators (a.k.a. "reviewers") on a case-by-case basis, appropriate for each paper or project being assessed. This is dependent on expertise, the researcher's interest, and a lack of conflict of interest.

Time commitment: Case-by-case basis. For each evaluation, here are some guidelines for the amount of time to spend.

Compensation: We pay a minimum of $200 (updated Aug. 2024) for a prompt and complete evaluation, $400 for experienced evaluators. We offer additional prizes and incentives, and are committed to an average compensation of at least $450 per evaluator. See here for more details.

Who we are looking for: We are putting together a list of people interested in being an evaluator and doing paid referee work for The Unjournal. We generally prioritize the pool of evaluators who signed up for our database before reaching out more widely.

Interested? Please fill out this form (about 3–5 min, same form for all roles or involvement).

Projects and papers

We are looking for high-quality, globally pivotal research projects to evaluate, particularly those embodying open science practices and innovative formats. We are putting out a call for relevant research. Please suggest research here. (We offer bounties and prizes for useful suggestions – see note.) For details of what we are looking for, and some potential examples, see this post and accompanying links.

You can also put forward your own work.

Contact us

Note: This is under continual refinement; see our policies for more details.

Submit your work to be eligible for our “Unjournal: Impactful Research Prize” and a range of other benefits including the opportunity for credible public evaluation and feedback.

First-prize winners will be awarded $2500, and the runner-ups will receive $1000.

Note: these are the minimum amounts; we will increase these if funding permits.

Prize winners will have the opportunity (but not the obligation) to present their work at an online seminar and prize ceremony co-hosted by The Unjournal, Rethink Priorities, and EAecon.

The Unjournal, with funding from the Long Term Future Fund and the Survival and Flourishing Fund, organizes and funds public-journal-independent feedback and evaluation. We focus on research that is highly relevant to global priorities, especially in economics, social science, and impact evaluation, and aim to expand this widely. We encourage better research by making it easier for researchers to get feedback and credible ratings on their work.

We aim to publicly evaluate 15 papers (or projects) within our pilot year. This award will honor researchers doing robust, credible, transparent work with a global impact. We especially encourage the submission of research in "open" formats such as hosted dynamic documents (Quarto, R-markdown, Jupyter notebooks, etc.).

The research will be chosen by our management team for public evaluation by 2–3 carefully selected, paid reviewers based on an initial assessment of a paper's methodological strength, openness, clarity, relevance to global priorities, and the usefulness of further evaluation and public discussion. We sketch out these criteria here.

All evaluations, including quantitative ratings, will be made public by default; however, we will consider "embargos" on this for researchers with sensitive career concerns (the linked form asks about this). Note that submitting your work to The Unjournal does not imply "publishing" it: you can submit it to any journal before, during, or after this process.

If we choose not to send your work out to reviewers, we will try to at least offer some brief private feedback (please on this).

All work evaluated by The Unjournal will be eligible for the prize. Engagement with The Unjournal, including responding to evaluator comments, will be a factor in determining the prize winners. We also have a slight preference for giving at least one of the awards to an early-career researcher, but this need not be determinative.

Our management team and advisory board will vote on the prize winners in light of the evaluations, with possible consultation of further external expertise.

Deadline: Extended until 5 December (to ensure eligibility).

Note: In a subsection below, Recap: submissions, we outline the basic requirements for submissions to The Unjournal.

How we chose the research prize winners (2023)

The prize winners for The Unjournal's Impactful Research Prize were selected through a multi-step, collaborative process involving both the management team and the advisory board. The selection was guided by several criteria, including the quality and credibility of the research, its potential for real-world impact, and the authors' engagement with The Unjournal's evaluation process.

1. Initial Evaluation: All papers that were evaluated by The Unjournal were eligible for the prize. The discussion, evaluations, and ratings provided by external evaluators played a significant role in the initial shortlisting.

2. Management and Advisory Board Input: Members of the management committee and advisory board were encouraged to write brief statements about papers they found particularly prize-worthy.

3. Meeting and Consensus: A "prize committee" meeting was held with four volunteers from the management committee to discuss the shortlisted papers and reach a consensus. The committee considered both the papers and the content of the evaluations in detail. Members of the committee allocated a total of 100 points among the 10 paper candidates. We used this to narrow down a shortlist of five papers.

4. Point Voting: The above shortlist and the notes from the accompanying discussion were shared with all management committee and advisory board members. Everyone in this larger group was invited to allocate up to 100 points among the shortlisted papers (and asked to allocate fewer points if they were less familiar with the papers and evaluations).

5. Special Considerations: We decided that at least one of the winners had to be a paper submitted by the authors or one where the authors substantially engaged with The Unjournal's processes. However, this constraint did not prove binding. Early-career researchers were given a slight advantage in our consideration.

6. Final Selection: The first and second prizes were given to the papers with the first- and second-most points, respectively.

This comprehensive approach aimed to ensure that the prize winners were selected in a manner that was rigorous, fair, and transparent, reflecting the values and goals of The Unjournal.

See also:

The Unjournal was founded by , who maintains this wiki/GitBook and other resources.

Management Committee

( on terminology)

See description under .

• , Founder and Co-director

• , Interdisciplinary Researcher at ; Co-director

• , Social Scientist and Associate Professor in the Guelph Institute of Development Studies and Department of Political Science at the University of Guelph, Canada

• , Economics PhD student at the University of California, Merced

• , Research Author at the Department of Psychology, (India)

• , Senior Research Fellow, Institute of Environment & Sustainability, Lee Kuan Yew of School of Public Policy, National University of Singapore

• , Research Scientist (fellow) at (South Africa)

• , Research Author at the Department of Economics at (India)

Advisory board

Field Specialists

Staff, contractors, and consultants

Other people and initiatives we are in touch with

Proposed projects

We are planning to hire 3–7 researchers for a one-off paid project.

There are two opportunities: Contracted Research (CR) and Independent Projects (IP).

Project Outline

• What specific research themes in economics, policy, and social science are most important for global priorities?

• What projects and papers are most in need of further in-depth public evaluation, attention, and scrutiny?

• Where does "Unjournal-style evaluation" have the potential to be one of the most impactful uses of time and money? By impactful, we mean in terms of some global conception of value (e.g., the well-being of living things, the survival of human values, etc.).

This is an initiative that aims to identify, summarize, and conduct an in-depth evaluation of the most impactful themes in economics, policy, and social science to answer the above questions. Through a systematic review of selected papers and potential follow-up with authors and evaluators, this project will enhance the visibility, understanding, and scrutiny of high-value research, fostering both rigorous and impactful scholarship.

Contracted Research (CR) This is the main opportunity, a unique chance to contribute to the identification and in-depth evaluation of impactful research themes in economics, policy, and social science. We’re looking for researchers and research users who can commit a (once-off) 15–20 hours. CR candidates will:

• Summarize a research area or theme, its status, and why it may be relevant to global priorities (~4 hours).

  • We are looking for fairly narrow themes. Examples might include:

    • The impact of mental health therapy on well-being in low-income countries.

    • The impact of cage-free egg regulation on animal welfare.

    • Public attitudes towards AI safety regulation.

• Identify a selection of papers in this area that might be high-value for UJ evaluation (~3 hours).

  • Choose at least four of these from among NBER/"top-10 working paper" series (or from work submitted to the UJ – we can share – or from work where the author has expressed interest to you).

• For a single paper, or a small set of these papers (or projects) (~6 hours)

  • Read the paper fairly carefully and summarize it, explaining why it is particularly relevant.

  • Discuss one or more aspects of the paper that need further scrutiny or evaluation.

  • Identify 3 possible evaluators, and explain why they might be particularly relevant to evaluate this work. (Give a few sentences we could use in an email to these evaluators).

  • Possible follow-up task: email and correspond with the authors and evaluators (~3 hours).

We are likely to follow up on your evaluation suggestions. We also may incorporate your writing into our web page and public posts; you can choose whether you want to be publicly acknowledged or remain anonymous.

Independent Projects (IP)

We are also inviting applications to do similar work as an “Independent Project” (IP), a parallel opportunity designed for those eager to engage but not interested in working under a contract, or not meeting some of the specific criteria for the Contracted Research role. This involves similar work to above.

If you are accepted to do an IP, we will offer some mentoring and feedback. We will also offer prize rewards/bounties for particularly strong IP work. We will also consider working with professors and academic supervisors on these IP projects, as part of university assignments and dissertations.

You can apply to the CR and IP positions together; we will automatically consider you for each.

Get Involved!

Note on .

Initiative: ‘independent evaluations’

is seeking academics, researchers, and students to submit structured evaluations of the most impactful research . Strong evaluations will be posted or linked on our , offering readers a perspective on the implications, strengths, and limitations of the research. These evaluations can be submitted using for academic-targeted research or for ; evaluators can publish their name or maintain anonymity; we also welcome collaborative evaluation work. We will facilitate, promote, and encourage these evaluations in several ways, described below.

Who should do these evaluations?

We are particularly looking for people with research training, experience, and expertise in quantitative social science and statistics including cost-benefit modeling and impact evaluation. This could include professors, other academic faculty, postdocs, researchers outside of academia, quantitative consultants and modelers, PhD students, and students aiming towards PhD-level work (pre-docs, research MSc students etc.) But anyone is welcome to give this a try — when in doubt, piease go for it.

We are also happy to support collaborations and group evaluations. There is a good track record for this — see: “”, ASAPBio’s, and for examples in this vein. We may also host live events and/or facilitate asynchronous collaboration on evaluations

Instructors/PhD, MRes, Predoc programs: We are also keen to work with students and professors to integrate ‘independent evaluation assignments’ (aka ‘learn to do peer reviews’) into research training.

Why should you do an evaluation?

Your work will support The Unjournal’s core mission — improving impactful research through journal-independent public evaluation. In addition, you’ll help research users (policymakers, funders, NGOs, fellow researchers) by providing high quality detailed evaluations that rate and discuss the strengths, limitations, and implications of research.

Doing an independent evaluation can also help you. We aim to provide feedback to help you become a better researcher and reviewer. We’ll also give prizes for the strongest evaluations. Lastly, writing evaluations will help you build a portfolio with The Unjournal, making it more likely we will commission you for paid evaluation work in the future.

Which research?

We focus on rigorous, globally-impactful research in quantitative social science and policy-relevant research. (See for details.) We’re especially eager to receive independent evaluations of:

1. Research we publicly prioritize: see our we've prioritized or evaluated. ()

2. Research we previously evaluated (see , as well as )

3. Work that other people and organizations suggest as having high potential for impact/value of information (also see)

What sort of ‘evaluations’ and what formats?

The Unjournal’s structured evaluation forms: We encourage evaluators to do these using either:

How will The Unjournal engage?

1. Posting and signal-boosting

2. Offering incentives

Bounties: We will offer prizes for the ‘most valuable independent evaluations’.

All evaluation submissions will be eligible for these prizes and “grandfathered in” to any prizes announced later. We will announce and promote the prize winners (unless they opt for anonymity).

Evaluator pool: People who submit evaluations can elect to join our evaluator pool. We will consider and (time-permitting) internally rate these evaluations. People who do the strongest evaluations in our focal areas are likely to be commissioned as paid evaluators for The Unjournal.

3. Providing materials, resources and guidance/feedback

4. Partnering with academic institutions

We are reaching out to PhD programs and pre-PhD research-focused programs. Some curricula already involve “mock referee report” assignments. We hope professors will encourage their students to do these through our platform. In return, we’ll offer the incentives and promotion mentioned above, as well as resources, guidance, and some further feedback

5. Fostering a positive environment for anonymous and signed evaluations

We want to preserve a positive and productive environment. This is particularly important because we will be accepting anonymous content. We will take steps to ensure that the system is not abused. If the evaluations have an excessively negative tone, have content that could be perceived as personal attacks, or have clearly spurious criticism, we will ask the evaluators to revise this, or we may decide not to post or link it.

How does this benefit The Unjournal and our mission?

1. Crowdsourced feedback can add value in itself; encouraging this can enable some public evaluation and discussion of work that The Unjournal doesn’t have the bandwidth to cover

2. Improving our evaluator pool and evaluation standards in general.

   1. Students and ECRs can practice and (if possible) get feedback on independent evaluations

   2. They can demonstrate their ability this publicly, enabling us to recruit and commission the strongest evaluators

3. Examples will help us build guidelines, resources, and insights into ‘what makes an evaluation useful’.

4. This provides us opportunities to engage with academia, especially in Ph.D programs and research-focused instruction.

About The Unjournal (unfold)

I was in academia for about 20 years (PhD Economics, UC Berkeley; Lecturer, University of Essex; Senior Lecturer, University of Exeter). I saw how the journal system was broken.

• Academics constantly complain about it (but don't do anything to improve it).

• Most conversations are not about research, but about 'who got into what journal' and 'tricks for getting your paper into journals'

• Open science and replicability are great, and dynamic documents make research a lot more transparent and readable. But these goals and methods are very hard to apply within the traditional journal system and its 'PDF prisons'.

Now I'm working outside academia and can stick my neck out. I have the opportunity to help fix the system. I work with research organizations and large philanthropists involved with effective altruism and global priorities. They care about the results of research in areas that are relevant to global priorities. They want research to be reliable, robust, reasoning-transparent, and well-communicated. Bringing them into the equation can change the game.

Management committee members

Activities of those on the management committee may involve a combination of the following (although you can choose your focus):

• Contributing to the decision-making process regarding research focus, reviewer assignment, and prize distribution.

• Collaborating with other committee members on the establishment of rules and guidelines, such as determining the metrics for research evaluation and defining the mode of assessment publication.

• Helping plan The Unjournal’s future path.

• Helping monitor and prioritize research for The Unjournal to evaluate (i.e., acting as a ; see further discussion below). Acting as an for research in your area.

Time commitment: A minimum of 15–20 hours per year.

Compensation: We have funding for a $57.50 per hour honorarium for the first 20 hours, with possible compensation . will be further compensated (at roughly $300–$450 per paper).

Who we are looking for: All applicants are welcome. We are especially interested in those involved in global priorities research (and related fields), policy research and practice, open science and meta-science, bibliometrics and scholarly publishing, and any other academic research. We want individuals with a solid interest in The Unjournal project and its goals, and the ability to meet the minimal time commitment. Applying is extremely quick, and those not chosen will be considered for other roles and work going forward.

Advisory board (AB) members

Beyond direct roles within The Unjournal, we're building a larger, more passive advisory board to be part of our network, to offer occasional feedback and guidance. There is essentially no minimum time commitment for advisory board members—only opportunities to engage. We sketch some of the expectations in the fold below.

Field specialists (FS), Unjournal Research Affiliates (URA)

FSs will focus on a particular area of research, policy, or impactful outcome. They will keep track of new or under-considered research with potential for impact and explain and assess the extent to which The Unjournal can add value by commissioning its evaluation. They will "curate" this research and may also serve as evaluation managers for this work.

Time commitment: There is no specific time obligation—only opportunities to engage. We may also consult you occasionally on your areas of expertise. Perhaps 1–4 hours a month is a reasonable starting expectation for people already involved in doing or using research, plus potential additional paid assignments.

Who we are looking for: For the FS roles, we are seeking active researchers, practitioners, and stakeholders with a strong publication record and/or involvement in the research and/or research-linked policy and prioritization processes. For the AB, also people with connections to academic, governmental, or relevant non-profit institutions, and/or involvement in open science, publication, and research evaluation processes. People who can offer relevant advice, experience, guidance, or help communicate our goals, processes, and progress.

Unjournal Research Affiliates (URA)

Unjournal Research Affiliates will be expected to

1. Join our Slack and Coda, but there is no expectation to check in regularly.

2. Suggest 2-3 papers per term (Spring, Summer, Autumn), and explain why these have potential for impact. This comes with modest compensation; see above.

3. Give the 2nd opinion (assessment) for 3 papers per term.

4. Vote on 5-10 papers per term; check in to vote 1x per month or more

5. Understand and be able to explain The Unjournal to your colleagues/on social media

6. Join at least one (one hour) FS or Unjournal general meeting per year.

Evaluation management is not expected, but we may may consult URAs for occasional help in suggesting evaluators, with potential compensation.

Contact us

In addition to the "work roles," we are looking to engage researchers, research users, meta-scientists, and people with experience in open science, open access, and management of initiatives similar to The Unjournal.

We are continually looking to enrich our general team and board, including our #management-committee-members, #advisory-board-members-abm-and-field-specialists-fs These roles come with some compensation and incentives.

(Please see links and consider submitting an expression of interest).

These are principally not research roles, but familiarity with research and research environments will be helpful, and there is room for research involvement depending on the candidate’s interest, background, and skills/aptitudes.

There are currently one such role:

#h.htvmrldrqnhn (As of November 2023, still seeking freelancers)

Further note: We previously considered a “Management support and administrative professional” role. We are not planning to hire for this role currently. Those who indicated interest will be considered for other roles.

Express interest here.

Communications, Writing, and Public Relations Specialist

As of November 2023, we are soliciting applications for freelancers with skills in particular areas

The Unjournal is looking to work with a proficient writer who is adept at communicating with academics and researchers (particularly in economics, social science, and policy), journalists, policymakers, and philanthropists. As we are in our early stages, this is a generalist role. We need someone to help us explain what The Unjournal does and why, make our processes easy to understand, and ensure our outputs (evaluations and research synthesis) are accessible and useful to non-specialists. We seek someone who values honesty and accuracy in communication; someone who has a talent for simplifying complex ideas and presenting them in a clear and engaging way.

The work is likely to include:

1. Promotion and general explanation

   • Spread the word about The Unjournal, our approach, our processes, and our progress in press releases and short pieces, as well as high-value emails and explanations for a range of audiences

   • Make the case for The Unjournal to potentially skeptical audiences in academia/research, policy, philanthropy, effective altruism, and beyond

2. Keeping track of our progress and keeping everyone in the loop

   • Help produce and manage our external (and some internal) long-form communications

   • Help produce and refine explanations, arguments, and responses

   • Help provide reports to relevant stakeholders and communities

3. Making our rules and processes clear to the people we work with

   • Explain our procedures and policies for research submission, evaluation, and synthesis; make our systems easy to understand

   • Help us build flexible communications templates for working with research evaluators, authors, and others

4. Other communications, presentations, and dissemination

   • Write and organize content for grants applications, partnership requests, advertising, hiring, and more

   • Potentially: compose non-technical write-ups of Unjournal evaluation synthesis content (in line with interest and ability)

Most relevant skills, aptitudes, interests, experience, and background knowledge:

• Understanding of The Unjournal project

• Strong written communications skills across a relevant range of contexts, styles, tones, and platforms (journalistic, technical, academic, informal, etc.)

• Familiarity with academia and research processes and institutions

• Familiarity with current conversations and research on global priorities within government and policy circles, effective altruism, and relevant academic fields

• Willingness to learn and use IT, project management, data management, web design, and text-parsing tools (such as those mentioned below), with the aid of GPT/AI chat

Further desirable skills and experience:

• Academic/research background in areas related to The Unjournal’s work

• Operations, administrative, and project management experience

• Experience working in a small nonprofit institution

• Experience with promotion and PR campaigns and working with journalists and bloggers

Proposed terms:

• Project-based contract "freelance" work

• $30–$55/hour USD (TBD, depending on experience and capabilities). Hours for each project include some onboarding and upskilling time.

• Our current budget can cover roughly 200 hours of this project work. We hope to increase and extend this (depending on our future funding and expenses).

• This role is contract-based and supports remote and international applicants. We can contract people living in most countries, but we cannot serve as an immigration sponsor.

Express your interest here.

The potential roles discussed below combine research-linked work with operations and administrative responsibilities. Overall, this may include some combination of:

• Assisting and guiding the process of identifying strong and potentially impactful work in key areas, explaining its relevance, its strengths, and areas warranting particular evaluation and scrutiny

• Interacting with authors, recruiting, and overseeing evaluators

• Synthesizing and disseminating the results of evaluations and ratings

• Aggregating and benchmarking these results

• Helping build and improve our tools, incentives, and processes

• Curating outputs relevant to other researchers and policymakers

• Doing "meta-science" work

Possible role: Research and Evaluation Specialist (RES)

Express your interest here. (Nov. 2023: Note, we can not guarantee that we will be hiring for this role, because of changes in our approach.)

TLDR: See #in-a-nutshell

Podcasts, presentations, and video

Journal independent evaluation and The Unjournal

EA Anywhere (Youtube) – bridging the gap between EA and academia

• See slide deck (Link: bit.ly/unjourrnalpresent; offers comment access)

• Presentation summarized with time-stamped hyperlinks (+~AI generated content)

ReproducibiliTea podcast

Slide decks

Presentation for EA Anywhere, online event, 5 Nov. 2023 1-2pm ET

(Link: bit.ly/unjourrnalpresent; offers comment access)

"Slaying the journals": Google doc

Earlier discussion document, aimed at EA/global priorities, academic, and open-science audiences [link]

"Moving science beyond ... static journals" ... How EA/nonprofits can help

Moving science beyond closed, binary, static journals; a proposed alternative; how the "Effective Altruist" and nontraditional nonprofit sector can help make this happen

• 2021 A shorter outline posted on onscienceandacademia.org

EA forum posts

Did you just write a brilliant peer review for an economics (or social science, policy, etc.) journal? Your work should not be wasted, there should be a way to share your insights and get credit!

Consider transforming these insights into a public "independent evaluation" for . This will benefit the community and help make research better and more impactful. And we can share your work and provide you feedback. This will help you build a portfolio with The Unjournal, making it more likely we'll hire you for paid work and compensate you at the higher rate. And we offer prizes for the best work.

You can do this either anonymously or sign your name.

To say this in a fancier way:

Journal peer review is critical for assessing and improving research, but too often these valuable discussions remain hidden behind closed doors. By publishing a version of your review, you can: (1) Amplify the impact of your reviewing efforts by contextualizing the research for a broader audience, (2) Facilitate more transparent academic discussions around the strengths and limitations of the work, (3) Get public recognition for your peer review contributions, which are often unseen and unrewarded (4) Reduce overall reviewing burdens by allowing your assessment to be reused, (5) Support a culture of open scholarship by modeling constructive feedback on public research

Considerations: The reviewer "owns the review", subject to constraints and norms

According to a COPE Discussion document: Who “owns” peer reviews (emphasis added)

While the depth of commentary may vary greatly among reviews, given the minimal thresholds set by copyright law, it can be presumed that most reviews meet the requirements for protection as an “original work of authorship”. As such, in the absence of an express transfer of copyright or a written agreement between the reviewer and publisher establishing the review as a “work for hire”, it may be assumed that, by law, the reviewer holds copyright to their reviewer comments and thus is entitled to share the review however the reviewer deems fit...

The COPE council notes precisely the benefits we are aiming to unlock. They mention an 'expectation of confidentiality' that seems incompletely specified.

For example, reviewers may wish to publish their reviews in order to demonstrate their expertise in a subject matter and to contribute to their careers as a researcher. Or they may see publication of their reviews as advancing discourse on the subject and thus acting for the benefit of science as a whole. Nevertheless, a peer reviewer’s comments are significantly different from many other works of authorship in that they are expressly solicited as a work product by a journal and—whatever the peer review model—are subject to an expectation of confidentiality. However, without an express agreement between the journal and the reviewer, it is questionable whether such obligation of confidentiality should be considered to apply only until a final decision is reached on the manuscript, or to extend indefinitely.

Permission and journals' constraints

Several journals explicitly agree that reviewers are welcome to publish the content of their reviews, with some important caveats. The Publish Your Reviews initiative gathered public statements from several journals and publishers confirming that they support reviewers posting their comments externally. However, they generally ask reviewers to remove any confidential information before sharing their reviews. This includes: the name of the journal, the publication recommendation (e.g., accept, revise, or reject), and any other details the journal or authors considered confidential, such as unpublished data.

For these journals, we are happy to accept and share/link the verbatim content as part of an independent Unjournal evaluation.

But even for journals that have not signed onto this, as the COPE mentioned Your peer review is your intellectual property, it is not owned by the journal!

There may be some terms and conditions you agreed to as part of submitting a referee report. Please consult these carefully.

However, you are still entitled to share your own expert opinions on publicly-shared research. You may want to rewrite the review somewhat. You should make it clear that it refers to the publicly-shared (working paper/preprint) version of the research, not the one the journal shared with you in confidence. As above, you should probably not mention the journal name, the decision, or any other sensitive information. You don't even need to mention that you did review the paper for a journal.

Even if a journal considers the specific review confidential, this doesn't prevent the reviewer from expressing their independent assessment elsewhere.

Make a difference

As an expert reviewer, you have unique insights that can improve the quality and impact of research. Making your assessment available through The Unjournal amplifies the reach and value of your efforts. You can publish evaluations under your name or remain anonymous.

Ready to make your peer reviews work harder for science? Consider submitting an independent evaluation, for recognition, rewards, and to improve research. Contact us anytime at contact@unjournal.org for guidance... We look forward to unlocking your valuable insights!

What do we offer? How does it improve upon traditional academic review/publishing?

: The Unjournal's process reduces the high costs and "gaming" associated with standard journal publication mechanisms.

: We promote research replicability and robustness in line with the RRC agenda.

: We prioritize impactful work. Expert evaluators focus on reliability, robustness, and usefulness. This fosters a productive bridge between high-profile mainstream researchers and global-impact-focused organizations, researchers, and practitioners.

: We open up the evaluation process, making it more timely and transparent, and providing valuable public metrics and feedback for the benefit of authors, other researchers, and policymakers who may want to use the research..

: By separating evaluation from journal publication, free research from the static 'PDF prisons'. This enables "dynamic documents/notebooks" that boost transparency and replicability, improved research communication through web-based formats. It also facilitates "living projects": research that can continuously grow, improving in response to feedback and incorporating new data and methods in the same environment.

March 25 2024: Workshop: Innovations in Research Evaluation, Replicability, and Impact

Research evaluation is changing: New approaches go beyond the traditional journal model, promoting transparency, replicability, open science, open access, and global impact. You can be a part of this.

Join us on March 25 for an interactive workshop, featuring presentations from Macie Daley (Center for Open Science), (The Unjournal), (UC Santa Barbara), and The Unjournal’s Impactful Research Prize and Evaluator Prize winners. Breakout discussions, Q&A, and interactive feedback sessions will consider innovations in open research evaluation, registered revisions, research impact, and open science methods and career opportunities.

The event will be held fully online on Zoom, on March 25 from 9AM- 11:30 AM (EST) and 9:30 PM - Midnight (EST) to accommodate a range of time zones. UTC: 25-March 1pm-3:30pm and 26-March 1:30am-4am. The event is timetabled: feel free to participate in any part you wish.

See the for all details, and to registr.

Jan 2024: Impactful Research and Evaluation Prizes winners announced

Aug. 30, 2023: "Pilot's done, what has been won (and learned)?"

Pilot = completed!

With the completed set of evaluations of and ,” our pilot is complete:

• 10 research papers evaluated

• 21 evaluations

• 5 author responses

Following up on the pilot: prizes and seminars

Following this, we are considering holding an online workshop (that will include a ceremony for the awarding of prizes). Authors and (non-anonymous) evaluators will be invited to discuss their work and take questions. We may also hold an open discussion and Q&A on The Unjournal and our approach. We aim to partner with other organizations in academia and in the impactful-research and open-science spaces. If this goes well, we may make it the start of a regular thing.

Other planned follow-ups from the pilot

Our pilot yielded a rich set of data and learning-by-doings. We plan to make use of this, including . . .

• synthesizing and reporting on evaluators' and authors' comments on our process; adapting these to make it better;

• analyzing the evaluation metrics for patterns, potential biases, and reliability measures;

• "aggregating expert judgment" from these metrics;

• tracking future outcomes (traditional publications, citations, replications, etc.) to benchmark the metrics against; and

• drawing insights from the evaluation content, and then communicating these (to policymakers, etc.).

The big scale-up

Evaluating more research: prioritization

2. • discuss and report on the state of research in their areas, including where and when relevant research is posted publicly, and in what state;

   • the potential for Unjournal evaluation of this work as well as when and how we should evaluate it, considering potential variations from our basic approach; and

   • how to prioritize work in this area for evaluation, reporting general guidelines and principles, and informing the aforementioned frameworks.

   Most concretely, the field teams will divide up the space of research work to be scoped and prioritized among the members of the teams.

Growing The Unjournal Team

Our previous call for field specialists is still active. We received a lot of great applications and strong interest, and we plan to send out invitations soon. But the door is still open to express interest!

Improving the evaluation process and metrics

We don't want to reinvent the wheel (unless we can make it a bit more round). We will be informed by previous work, such as:

• existing research into the research evaluation process, and on expert judgment elicitation and aggregation;

• practices from projects like RepliCATS/IDEAS, PREreview BITSS Open Policy Analysis, the “Four validities” in research design, etc.; and

Of course, our context and goals are somewhat distinct from the initiatives above.

We also aim to consult potential users of our evaluations as to which metrics they would find most helpful.

(A semi-aside: The choice of metrics and emphases could also empower efforts to encourage researchers to report policy-relevant parameters more consistently.)

We aim to bring a range of researchers and practitioners into these questions, as well as engaging in public discussion. Please reach out.

"Spilling tea"

I hope to do more of this sort of promotion: I'm happy to go on podcasts and other forums and answer questions about The Unjournal, respond to doubts you may have, consider your suggestions and discuss alternative initiatives.

MailChimp link: Sign up below to get these progress updates in your inbox about once per fortnight, along with opportunities to give your feedback.

Progress notes since last update

Hope these updates are helpful. Let me know if you have suggestions.

Post-pilot goals

See the vision and broad plan presented (and embedded below), updated August 2023.

Pilot targets

Building research "unjournal"

Setup and team

Status: Mostly completed/decided for pilot phase

Create a set of rules for "submission and management"

• Which projects enter the review system (relevance, minimal quality, stakeholders, any red lines or "musts")
• How projects are to be submitted

• How reviewers are to be assigned and compensated

Status: Mostly completed/decided for pilot phase; will review after initial trial

Rules for reviews/assessments

• To be done on the chosen open platform (Kotahi/Sciety) unless otherwise infeasible (10 Dec 2022 update)

• Share, advertise, promote this; have efficient meetings and presentations

  • Establish links to all open-access bibliometric initiatives (to the extent feasible)

• Harness and encourage additional tools for quality assessment, considering cross-links to prediction markets/Metaculus, to coin-based 'ResearchHub', etc.

Status: Mostly completed/decided for pilot phase; will review after the initial trial

Further steps

Status: We are still working with Google Docs and building an external survey interface. We plan to integrate this with PubPub over the coming months (August/Sept. 2023)

Unjournal evaluations aim to support the "Reproducibility/Robustness-Checking" (RRC) agenda. We are directly engaging with the Institute for Replication (I4R) and the repliCATS project (RC), and building connections to Replication Lab/TRELiSS and Metaculus.

We will support this agenda by:

1. Promoting data and code sharing: We request pre-print authors to share their code and data, and reward them for their transparency.

2. Promoting 'Dynamic Documents' and 'Living Research Projects': Breaking out of "PDF prisons" to achieve increased transparency.

3. Encouraging detailed evaluations: Unjournal evaluators are asked to:

   • highlight the key/most relevant research claims, results, and tests;

   • propose possible robustness checks and tests (RRC work); and

   • make predictions for these tests.

4. Implementing computational replication and robustness checking: We aim to work with I4R and other organizations to facilitate and evaluate computational replication and robustness checking.

5. Advocating for open evaluation: We prioritize making the evaluation process transparent and accessible for all.

Research credibility

While the replication crisis in psychology is well known, economics is not immune. Some very prominent and influential work has blatant errors, depends on dubious econometric choices or faulty data, is not robust to simple checks, or uses likely-fraudulent data. Roughly 40% of experimental economics work fail to replicate. Prominent commenters have argued that the traditional journal peer-review system does a poor job of spotting major errors and identifying robust work.

Supporting the RRC agenda through Unjournal evaluations

My involvement with the SCORE replication market project shed light on a key challenge (see Twitter posts): The effectiveness of replication depends on the claims chosen for reproduction and how they are approached. I observed that it was common for the chosen claim to miss the essence of the paper, or to focus on a statistical result that, while likely to reproduce, didn't truly convey the author's message.

Simultaneously, I noticed that many papers had methodological flaws (for instance, lack of causal identification or the presence of important confounding factors in experiments). But I thought that these studies, if repeated, would likely yield similar results. These insights emerged from only a quick review of hundreds of papers and claims. This indicates that a more thorough reading and analysis could potentially identify the most impactful claims and elucidate the necessary RRC work.

Indeed, detailed, high-quality referee reports for economics journals frequently contain such suggestions. However, these valuable insights are often overlooked and rarely shared publicly. Unjournal aims to change this paradigm by focusing on three main strategies:

1. Identifying vital claims for replication:

   • We plan to have Unjournal evaluators help highlight key "claims to replicate," along with proposing replication goals and methodologies. We will flag papers that particularly need replication in specific areas.

   • Public evaluation and author responses will provide additional insight, giving future replicators more than just the original published paper to work with.

2. Encouraging author-assisted replication:

   • The Unjournal's platform and metrics, promoting dynamic documents and transparency, simplify the process of reproduction and replication.

   • By emphasizing replicability and transparency at the working-paper stage (Unjournal evaluations’ current focus), we make authors more amenable to facilitate replication work in later stages, such as post-traditional publication.

3. Predicting replicability and recognizing success:

   • We aim to ask Unjournal evaluators to make predictions about replicability. When these are successfully replicated, we can offer recognition. The same holds for repliCATS aggregated/IDEA group evaluations: To know if we are credibly assessing replicability, we need to compare these to at least some "replication outcomes."

   • The potential to compare these predictions to actual replication outcomes allows us to assess the credibility of our replicability evaluations. It may also motivate individuals to become Unjournal evaluators, attracted by the possibility of influencing replication efforts.

By concentrating on NBER papers, we increase the likelihood of overlap with journals targeted by the Institute for Replication, thus enhancing the utility of our evaluations in aiding replication efforts.

Dynamic Documents

By “Dynamic Documents” I mean papers/projects built with Quarto, R-markdown, or JuPyTer notebooks (the most prominent tools) that do and report the data analysis (as well as math/simulations) in the same space that the results and discussion are presented (with ‘code blocks’ hidden).

I consider some of the benefits of this format, particularly for EA-aligned organizations like Open Philanthropy: Benefits of Dynamic Documents

Living Research Projects

“Continually update a project” rather than start a “new extension paper” when you see what you could have done better.

The main idea is that each version is given a specific time stamp, and that is the object that is reviewed and cited. This is more or less already the case when we cite working papers/drafts/mimeos/preprints.

See #living-kaizend-research-projects, further discussing the potential benefits.

See also...

Rate and give feedback, don’t accept/reject

Claim: Rating and feedback is better than an ‘all-or-nothing’ accept/reject process. Although people like to say “peer review is not binary”, the consequences are.

“Publication in a top journal” is used as a signal and a measuring tool for two major purposes. First, policymakers, journalists, and other researchers look at where a paper is published to assess whether the research is credible and reputable. Second, universities and other institutions use these publication outcomes to guide hiring, tenure, promotion, grants, and other ‘rewards for researchers.’

Pursuing 'top publications' can be very time-consuming and risky for career academics

In the field of economics, it is not unusual for it to take years between the ‘first working paper’ that is publicly circulated and the final publication. During that time, the paper may be substantially improved, but it may not be known to nor accepted by practitioners. Meanwhile, it provides little or no career value to the authors.

As a result, we see three major downsides:

1. Time spent gaming the system:

Researchers and academics spend a tremendous amount of time 'gaming' this process, at the expense of actually doing better research.

1. Randomness in outcomes, unnecessary uncertainty and stress

2. Wasted feedback, including reviewer's time

Time spent gaming the system

I (Reinstein) have been in academia for about 20 years. Around the departmental coffee pot and during research conference luncheons, you might expect us to talk about theories, methods, and results. But roughly half of what we talk about is “who got into which journal and how unfair it is”; “which journal should we be submitting our papers to?”; how long are their “turnaround times?”; “how highly rated are these journals?”; and so on. We even exchange tips on how to ‘sneak into these journals’.

There is a lot of pressure, and even bullying, to achieve these “publication outcomes” at the expense of careful methodology.

Randomness in outcomes

The current system can sideline deserving work due to unpredictable outcomes. There's no guarantee that the cream will rise to the top, making research careers much more stressful—even driving out more risk-averse researchers—and sometimes encouraging approaches that are detrimental to good science.

Wasted feedback

A lot of ‘feedback’ is wasted, including the reviewers' time. Some reviewers write ten-page reports critiquing the paper in great detail, even when they reject the paper. These reports are sometimes very informative and useful for the author and would also be very helpful for the wider public and research community to understand the nature of the debate and issues.

However, researchers often have a very narrow focus on getting the paper published as quickly and in as high-prestige a journal as possible. Unless the review is part of a 'Revise and Resubmit' that the author wants to fulfill, they may not actually put the comments into practice or address them in any way.

Of course, the reviews may be misinformed, mistaken, or may misunderstand aspects of the research. However, if the paper is rejected (even if the reviewer was positive about the paper), the author has no opportunity or incentive to respond to the reviewer. Thus the misinformed reviewer may remain in the dark.

The other side of the coin: a lot of effort is spent trying to curry favor with reviewers who are often seen as overly fussy and not always in the direction of good science.

The standard mode at top economics journals

Here's a rough sketch of the process and timings at top journals in economics. Hadevand et al report an average of over 24 months between initial submisson and final acceptance (and nearly three years until publication).

Our theory of change is shown above as a series of possible paths; we indicate what is arguably the most "direct" path in yellow. All of these paths begin with our setting up, funding, communicating, and incentivizing participation in a strong, open, efficient research evaluation system (in green, at the top). These processes all lead to impactful research being more in-depth, more reliable, more accessible, and more useful, and thus better informing decision-makers and leading to better decisions and outcomes (in green, at the bottom).

Highlighting some of the key paths:

(Yellow) Faster and better feedback on impactful research improves this work and better informs policymakers and philanthropists.

(Blue) Our processes and incentives will foster ties between (a) mainstream and prominent academic and policy researchers and (b) global-priorities or EA-aligned researchers. This will improve the rigor, credibility, exposure, and influence of previously "EA niche" work while helping mainstream researchers better understand and incorporate ideas, principles, and methods from the EA and rationalist research communities (such as counterfactual impact, cause-neutrality, reasoning transparency, and so on.) This process will also nudge mainstream academics towards focusing on impact and global priorities, and towards making their research and outputs more accessible and useable.

(Pink) The Unjournal’s more efficient, open, and flexible processes will become attractive to academics and stakeholders. As we become better at "predicting publication outcomes," we will become a replacement for traditional processes, improving research overall—some of which will be highly impactful research.

Detailed explanations of key paths

Rapid, informative, transparent feedback and evaluation to inform policymakers and researchers

Rigorous quantitative and empirical research in economics, business, public policy, and social science has the potential to improve our decision-making and enable a flourishing future. This can be seen in the research frameworks proposed by 80,000 Hours, Open Philanthropy, and The Global Priorities Institute (see discussions here). This research is routinely used by effective altruists working on global priorities or existential risk mitigation. It informs both philanthropic decisions (e.g., those influenced by GiveWell's Cost-Effectiveness Analyses, whose inputs are largely based on academic research) and national public policy. Unfortunately, the academic publication process is notoriously slow; for example, in economics, it routinely takes 2–6 years between the first presentation of a research paper and the eventual publication in a peer-reviewed journal. Recent reforms have sped up parts of the process by encouraging researchers to put working papers and preprints online.

However, working papers and preprints often receive at most only a cursory check before publication, and it is up to the reader to judge quality for themselves. Decision-makers and other researchers rely on peer review to judge the work’s credibility. This part remains slow and inefficient. Furthermore, it provides very noisy signals: A paper is typically judged by the "prestige of the journal it lands in"’ (perhaps after an intricate odyssey across journals), but it is hard to know why it ended up there. Publication success is seen to depend on personal connections, cleverness, strategic submission strategies, good presentation skills, and relevance to the discipline’s methods and theory. These factors are largely irrelevant to whether and how philanthropists and policymakers should consider and act on a paper’s claimed findings. Reviews are kept secret; the public never learns why a paper was deemed worthy of a journal, nor what its strengths and weaknesses were.

We believe that disseminating research sooner—along with measures of its credibility—is better.

We also believe that publicly evaluating its quality before (and in addition to) journal publication will add substantial additional value to the research output, providing:

1. a quality assessment (by experts in the field) that can decisionmakers and other researchers can read alongside the preprint, helping these users weigh its strengths and weaknesses and interpret its implications; and

2. faster feedback to authors focused on improving the rigor and impact of the work.

Various initiatives in the life sciences have already begun reviewing preprints. While economics took the lead in sharing working papers, public evaluation of economics, business, and social science research is rare. The Unjournal is the first initiative to publicly evaluate rapidly-disseminated work from these fields. Our specific priority: research relevant to global priorities.

Faster, better feedback; attractiveness to researchers and gatekeepers; improved research formats; and better and more useful research

The Unjournal’s open feedback should also be valuable to the researchers themselves and their research community, catalyzing progress. As the Unjournal Evaluation becomes a valuable outcome in itself, researchers can spend less time "gaming the journal system." Shared public evaluation will provide an important window to other researchers, helping them better understand the relevant cutting-edge concerns. The Unjournal will permit research to be submitted in a wider variety of useful formats (e.g., dynamic documents and notebooks rather than "frozen pdfs"), enabling more useful, replicable content and less time spent formatting papers for particular journals. We will also allow researchers to improve their work in situ and gain updated evaluations, rather than having to spin off new papers. This will make the literature more clear and less cluttered.

"Some of the main paths"

The Unjournal is delighted to announce the winners of our inaugural Impactful Research Prize. We are awarding our first prize to Takahiro Kubo (NIES Japan and Oxford University) and co-authors for their research titled "Banning wildlife trade can boost demand". The paper stood out for its intriguing question, the potential for policy impact, and methodological strength. We particularly appreciated the authors’ open, active, and detailed engagement with our evaluation process.

The second prize goes to Johannes Haushofer (NUS Singapore and Stockholm University) and co-authors for their work "The Comparative Impacts of Cash Transfers and a Psychotherapy Program on Psychological and Economic Wellbeing". Our evaluators rated this paper among the highest across a range of metrics. It was highly commended for its rigor, the importance of the topic, and the insightful discussion of cost-effectiveness.

We are recognizing exceptional evaluators for credible, insightful evaluations. Congratulations to Phil Trammell (Global Priorities Institute at the University of Oxford), Hannah Metzler (Complexity Science Hub Vienna), Alex Bates (independent researcher), and Robert Kubinec (NYU Abu Dhabi).

We would like to congratulate all of the winners on their contributions to open science and commitment to rigorous research. We also thank other authors who have submitted their work but have not been selected at this time - we received a lot of excellent submissions, and we are committed to supporting authors beyond this research prize.

Please see the full press release, as well as award details, below and linked here:

Open evaluation (and rating)

Traditional peer review is a closed process, with reviewers' and editors' comments and recommendations hidden from the public.

In contrast, a (along with authors' responses and evaluation manager summaries) are made public and easily accessible. We give each of these a separate DOI and work to make sure each enters the literature and bibliometric databases. We aim further to curate these, making it easy to see the evaluators' comments in the context of the research project (e.g., with sidebar/hover annotation).

Open evaluation is more useful:

• to other researchers and students (especially those early in their careers). Seeing the dialogue helps them digest the research itself and understand its relationship to the wider field. It helps them understand the strengths and weaknesses of the methods and approaches used, and how much agreement there is over these choices. It gives an inside perspective on how evaluation works.

• to people using the research, providing further perspectives on its value, strengths and weaknesses, implications, and applications.

Publicly posting evaluations and responses may also lead to higher quality and more reliability. Evaluators can choose whether or not they wish to remain anonymous; there are , but in either case, the fact that all the content is public may encourage evaluators to more fully and transparently express their reasoning and justifications. (And where they fail to do so, readers of the evaluation can take this into account.)

The fact that we are asking for evaluations and ratings of all the projects in our system—and not using "accept/reject"—should also drive more careful and comprehensive evaluation and feedback. At a traditional top-ranked journal, a reviewer may limit themselves to a few vague comments implying that the paper is "not interesting or strong enough to merit publication." This would not make sense within the context of The Unjournal.

More reliable, precise, and useful metrics

We do not "accept or reject" papers; we are evaluating research, not "publishing" it. But then, how do other researchers and students know whether the research is worth reading? How can policymakers know whether to trust it? How can it help a researcher advance their career? How can grantmakers and organizations know whether to fund more of this research?

As an alternative to the traditional measure of worth—asking, "what tier did a paper get published in?"—The Unjournal provides metrics: We ask evaluators to provide a specific set of ratings and predictions about aspects of the research, as well as aggregate measures. We make these public. We aim to synthesize and analyze these ratings in useful ways, as well as make this quantitative data accessible to meta-science researchers, meta-analysts, and tool builders.

Feel free to check out our and (these are our pilot metrics, we aim to refine these).

These metrics are separated into different categories designed to help researchers, readers, and users understand things like:

• How much can one believe the results stated by the authors (and why)?

• How relevant are these results for particular real-world choices and considerations?

• Is the paper written in a way that is clear and readable?

• How much does it advance our current knowledge?

We also request overall ratings and predictions . . . of the credibility, importance, and usefulness of the work, and to help benchmark these evaluations to each other and to the current "journal tier" system.

However, even here, the Unjournal metrics are also precise in a sense that "journal publication tiers" are not. There is no agreed-upon metric of exactly how journals rank (e.g., within economics' "top-5" or "top field journals"). More importantly, there is no clear measure of the relative quality and trustworthiness of the paper within particular journals.

In addition, there are issues of lobbying, career concerns, and timing, discussed elsewhere, which make the "tiers" system less reliable. An outsider doesn't know, for example:

• Was a paper published in a top journal because of a special relationship and connections? Was an editor trying to push a particular agenda?

• Was it published in a lower-ranked journal because the author needed to get some points quickly to fill their CV for an upcoming tenure decision?

In contrast, The Unjournal requires evaluators to give specific, precise, quantified ratings and predictions (along with an explicit metric of the evaluator's uncertainty over these appraisals).

Of course, our systems will not solve all problems associated with reviews and evaluations: power dynamics, human weaknesses, and limited resources will remain. But we hope our approach moves in the right direction.

Better feedback

Faster (public) evaluation

We want to reduce the time between when research is done (and a paper or other research format is released) and when other people (academics, policymakers, journalists, etc.) have a credible measure of "how much to believe the results" and "how useful this research is."

Here's how The Unjournal can do this.

3. Public evaluations and ratings: Rather than waiting years to see "what tier journal a paper lands in," the public can simply consult The Unjournal to find credible evaluations and ratings.

Journal-independent review allows work to be rated separately in different areas: theoretical rigor and innovation, empirical methods, policy relevance, and so on, with separate ratings in each category by experts in that area. As a researcher in the current system, I cannot both submit my paper and get public evaluation from (for example) JET and the Journal of Development Economics for a paper engaging both areas.

The Unjournal, and journal-independent evaluation, can enable this through

• commissioning a range of evaluators with expertise in distinct areas, and making this expertise known in the public evaluations;

• asking specifically for multiple dimensions of quantitative (and descriptive) feedback and ratings (see especially under our ); and

• allowing authors to gain evaluation in particular areas in addition to the implicit value of publication in specific traditional field journals.

We acknowledge the potential for "information hazards" when research methods, tools, and results become more accessible. This is of particular concern in the context of direct physical and biological science research, particularly in (although there is a case that specific ). ML/AI research may also fall into this category. Despite these potential risks, we believe that the fields we plan to cover—detailed above—do not primarily present such concerns.

In cases where our model might be extended to high-risk research—such as new methodologies contributing to terrorism, biological warfare, or uncontrolled AI—the issue of accessibility becomes more complex. We recognize that increasing accessibility in these areas might potentially pose risks.

While we don't expect these concerns to be raised frequently about The Unjournal's activities, we remain committed to supporting thoughtful discussions and risk assessments around these issues.

General FAQs

What is The Unjournal?

We organize and fund public, journal-independent feedback, rating, and evaluation of academic work. We focus on work that is highly relevant to global priorities, especially in economics, social science, and impact evaluation. We encourage better research by making it easier for researchers to get credible feedback. See here for more details.

Does The Unjournal charge fees?

No. The Unjournal does not charge any fees. In fact, unlike most traditional journals, we compensate evaluators for their time, and award prizes for strong work.

We are a nonprofit organization. We do not charge fees for access to our evaluations, and work to make them as open as possible. In future, we may consider sliding-scale fees for people submitting their work for Unjournal evaluation. If so, this would simply be to cover our costs and compensate evaluators. We are a nonprofit and will stay that way.

Is The Unjournal a journal?

No. We do not publish research. We just commission public evaluation and rating of relevant research that is already publicly hosted. Having your work evaluated in The Unjournal should not limit you from submitting it to any publication.

How is The Unjournal funded? Is it sustainable?

We have grants from philanthropists and organizations who are interested in our priority research areas. We hope that our work will provide enough value to justify further direct funding. We may also seek funding from governments and universities supporting the open-access agenda.

I have another question.

Sure! Please contact us at theunjournal@gmail.com.

Specific FAQs

For research authors

Suggesting and prioritizing research

Evaluation ('refereeing')

You can fill out this form to submit your work, or email contact@unjournal.org with questions .

How does The Unjournal's evaluation process work?

We generally seek two evaluators (aka 'reviewers') with research interests in your area and with complementary expertise. You, the author, can suggest areas you want to get feedback and suggestions on.

The evaluators write detailed and helpful evaluations, and submit them either "signed" or anonymously. Using our evaluation forms, they provide quantitative ratings on several dimensions, such as methods, relevance, and communication. They predict what journal tier the research will be published in, and what tier it should be published in. Here are the Guidelines for evaluators.

These evaluations and ratings are typically made public (see unjournal.pubpub.org), but you will have the right to respond before (or after) these are posted.

To consider your research we only need a link to a publicly hosted version of your work, ideally with a DOI. We will not "publish" your paper. The fact that we are handling your paper will not limit you in any way. You can submit it to any journal before, during, or after the process.

Can I ask for evaluations to be private?

You can request a conditional embargo by emailing us at contact@unjournal.org, or via the submission form. Please explain what sort of embargo you are asking for, and why. By default, we would like Unjournal evaluations to be made public promptly. However, we may make exceptions in special circumstances, particularly for very early-career researchers whose career prospects are particularly vulnerable.

If there is a vulnerable early-career researcher on the authorship team, we may allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite.

For further details on this, and some examples, see 'Conditional embargos' & exceptions.

Do you make any requests of authors?

We may ask for some of the below, but these are mainly optional. We aim to make the process very light touch for authors.

1. A link to a non-paywalled, hosted version of your work (in any format) which can be given a DOI.

2. Responses to clarification questions from our evaluators.

3. We may ask for a link to data and code, if possible. Note that our project is not principally about replication, and we are not insisting on this. However, sharing code, data, and materials is highly encouraged.

We also allow you to respond to evaluations, and we give your response its own DOI.

Why should I submit my work to The Unjournal? Why should I engage with them?

By submitting your research and engaging with public evaluation, you send a powerful public signal that you are confident in your work, open to constructive criticism, and motivated to seek the truth!

And there are further benefits to authors:

1. Substantive feedback will help you improve your research. Substantive and useful feedback is often very hard to get, especially for young scholars. It's hard to get anyone to read your paper – we can help!

2. Being evaluated by The Unjournal is a sign of impact. We select our research based on potential global relevance.

3. Ratings are markers of credibility for your work that could help your career.

4. The chance to publicly respond to criticism and correct misunderstandings.

5. Increasing the visibility of your work, which may lead to additional citations. We publicize our evaluations and the original papers on our social media feed, and occasionally in notebook and longer-form content.

6. A connection to the Open Science/Open Access and EA/Global Priorities communities. This may lead to grant opportunities, open up new ambitious projects, and attract strong PhD students to your research groups.

7. A reputation as an early adopter and innovator in open science.

8. Prizes: You may win an Impactful Research Prize (pilot) (publicity, reputation, and substantial financial prizes). These prizes are tied, in part, to your engagement with The Unjournal.

9. Undervalued or updated work: Your paper may have been "under-published". Perhaps there are a limited set of prestigious journals in your field. You now see ways you could improve the research. The Unjournal can help; we will also consider 'post-peer-review publication' evaluation.

10. Innovative formats: Journals typically require you to submit a LaTeX or MS Word file, and to use their fussy formats and styles. You may want to use tools like Quarto that integrate your code and data, allow you to present dynamic content, and enhance reproducibility. The Unjournal encourages this, and we can evaluate research in virtually any format.

Is it risky to have my work evaluated in The Unjournal?

There are risks and rewards to any activity, of course. Here we consider some risks you may weigh against the benefits mentioned above.

1. Exclusivity

2. Public negative feedback

Exclusivity

Some traditional journals might have restrictions on the public sharing of your work, and perhaps they might enforce these more strongly if they fear competition from The Unjournal.

However, The Unjournal is not exclusive. Having your paper reviewed and evaluated in The Unjournal will not limit your options; you can still submit your work to traditional journals.

Public negative feedback

Our evaluations are public. While there has been some movement towards open review, this is still not standard. Typically when you submit your paper, reviews are private. With The Unjournal, you might get public negative evaluations.

We think this is an acceptable risk. Most academics expect that opinions will differ about a piece of work, and everyone has received negative reviews. Thus, getting public feedback — in The Unjournal or elsewhere — should not particularly harm you or your research project.

Nonetheless, we are planning some exceptions for early-career researchers (see #can-i-ask-for-evaluations-to-be-private).

Unjournal evaluations should be seen as signals of research quality. Like all such signals, they are noisy. But submitting to The Unjournal shows you are confident in your work, and not afraid of public feedback.

I received the message "Your paper is being evaluated by The Unjournal". What does this mean?

Within our "Direct evaluation" track, The Unjournal directly chooses papers (from prominent archives, well-established researchers, etc.) to evaluate. We don't request authors' permission here.

As you can see in our evaluation workflow, on this track, we engage with authors especially at two points:

1. Informing the authors that the evaluation will take place, requesting they engage, and giving them the opportunity to request a conditional embargo or specific types of feedback.

   • Of particular interest: are we looking at the most recent version of the paper/project, or is there a further revised version we should be considering instead?

2. After the evaluations have been completed, the authors are given two weeks to respond, and have their response posted along with our 'package'. (Authors can also respond after we have posted the evaluations, and we will put their response in the same 'package', with a DOI etc.)

I received the message "The Unjournal can confirm we have received your submitted manuscript." What does this mean?

Once we receive unsolicited work from an author or authors, we keep it in our database and have our team decide on prioritization. If your paper is prioritized for evaluation, The Unjournal will notify you.

At present, we do not have a system to automatically share the status of author submissions with authors. We hope to put one in place. You can email us for clarification and updates.

You can still submit your work to any traditional journal.

What if we (the authors) have revised the paper since the version evaluated by The Unjournal?

The Unjournal aims to evaluate the most recent version of a paper. We reach out to authors to ensure we have the latest version at the start of the evaluation process.

If substantial updates are made to a paper during the evaluation process, authors are encouraged to share the updated version. We then inform our evaluators and ask if they wish to revise their comments.

If the evaluators can't or don't respond, we will note this and still link the newest version.

Authors are encouraged to respond to evaluations by highlighting major revisions made to their paper, especially those relevant to the evaluators' critiques. If authors are not ready to respond to evaluations, we can post a placeholder response indicating that responses and/or a revised version of the paper are forthcoming.

Re-evaluation: If authors and evaluators are willing to engage, The Unjournal is open to re-evaluating a revised version of a paper after publishing the evaluations of the initial version.

Why should I provide an "author's response" to evaluations?

We share evaluations with the authors and give them a chance to respond before we make the evaluations public (and again afterward, at any point). We add these to our evaluation packages on PubPub. Evaluation manager's (public) reports and our further communications incorporate the paper, the evaluations, and the authors' responses.

Authors' responses could bring several benefits...

1. Personally: a chance to correct misconceptions and explain their approach and planned steps. If you spot any clear errors or omissions, we give evaluators a chance to adjust their reports in response. Your response can also help others have a more accurate and positive view of the research. This includes the evaluators, as well as future journal referees and grant funders.

2. For research users, to get an informed balanced perspective on how to judge the work

3. For other researchers, to better understand the methodological issues and approaches. This can serve to start a public dialogue and discussion to help build the field and research agenda. Ultimately, we aim to facilitate a back-and-forth between authors, evaluators, and others.

What should an 'author response' look like?

Evaluations may raise substantive doubts and questions, and make some specific suggestions, and ask followup questions about (e.g.) data, context, or assumptions. There's no need to respond to every evaluator point. Only respond where you have something substantive: clarifying doubts, explaining the justification for your particular choices, and giving your thoughts on the suggestions (which will you incorporate, or not, and why?).

A well-written author response might have a clear narrative and group responses into themes.

Try to have a positive tone (no personal attacks etc.) but avoid formality, over-politeness, or flattery. Revise-and-resubmit responses at standard journals sometimes begin each paragraph with "thank you for this excellent suggestion". Feel free to skip this; we want to focus on the substance.

Examples: We've received several detailed and informative author responses, such as:

• Gerster and Epperson, “Willful Ignorance and Moral Behavior”

• Denkenberger et al on "Long term cost-effectiveness of resilient foods..."

• Trammell and Aschenbrenner, “Existential Risk and Growth”

1. Project submission, selection and prioritization

People and organizations submit their own research or suggest research they believe may be high-impact. The Unjournal also directly monitors key sources of research and research agendas. Our team then systematically prioritizes this research for evaluation. See the link below for further details.

2. Evaluation

1. We choose an evaluation manager for each research paper or project. They commission and compensate expert evaluators to rate and discuss the research, following our evaluation template and . The original research authors are given a chance to publicly respond before we post these evaluations. See the link below for further details.

3. Communicating results

We make all of this evaluation work public on , along with an evaluation summary. We create DOIs for each element and submit this work to scholarly search engines. We also present a summary and analysis of our .

We outline some further details in the link below.

Flowchart

See for a full 'flowchart' map of our evaluation workflow

Other approaches

We are also piloting several initiatives that involve a different process. See:

Submission/evaluation funnel

As we are paying evaluators and have limited funding, we cannot evaluate every paper and project. Papers enter our database through:

1. submission by authors;

2. our own searches (e.g., searching syllabi, forums, working paper archives, and white papers); and

3. s from other researchers, practitioners, and members of the public, and recommendations from . We have posted more detailed instructions for .

Our management team rates the suitability of each paper according to the criteria discussed below and .

Our procedures for identification and prioritization

We have followed a few procedures for finding and prioritizing papers and projects. In all cases, we require more than one member of our research-involved team (field specialist, managers, etc.) to support a paper before prioritizing it.

We are building a grounded systematic procedure with criteria and benchmarks. We also aim to give managers and field specialists some autonomy in prioritizing key papers and projects. As noted elsewhere, we are considering targets for particular research areas and sources.

See our basic process (as of Dec. 2023) for prioritizing work:

Authors' permission: sometimes required

Through October 2022: For the papers or projects at the top of our list, we contacted the authors and asked if they wanted to engage, only pursuing evaluation if agreed.

In our , we inform authors but do not request permission. For this track, we have largely focused on working papers.

Communicating: "editors'" process

Summary: why is it relevant and worth engaging with?

Why does it need (more) review? What are some key issues or claims to vet?

What are (some of) the authors’ main claims that are worth carefully evaluating? What aspects of the evidence, argumentation, methods, interpretation, etc., is the team unsure about? What particular data, code, proof, etc., would they like to see vetted? If it has already been peer-reviewed in some way, why do they think more review is needed?

To what extent is there author engagement?

How well has the author engaged with the process? Do they need particular convincing? Do they need help making their engagement with The Unjournal successful?

Our is quantitative work that informs , especially in . We want to see better research leading to better outcomes in the real world (see our '').

To reach these goals, we need to select "the right research" for evaluation. We want to choose papers and projects that are highly relevant, methodologically promising, and that will benefit substantially from our evaluation work. We need to optimize how we select research so that our efforts remain mission-focused and useful. We also want to make our process transparent and fair. To do this, we are building a coherent set of criteria and goals, and a specific approach to guide this process. We explore several dimensions of these criteria below.

High-level considerations for prioritizing research

When considering a piece of research to decide whether to commission it to be evaluated, we can start by looking at its general relevance as well as the value of evaluating and rating it.

1. Why is it relevant and worth engaging with?

We consider (and prioritize) the importance of the research to global priorities; its relevance to crucial decisions; the attention it is getting, the influence it is having; its direct relevance to the real world; and the potential value of the research for advancing other impactful work. We de-prioritize work that has already been credibly (publicly) . We also consider the fit of the research with our scope (social science, etc.), and the likelihood that we can commission experts to meaningfully evaluate it. As noted , some 'instrumental goals' (, , driving change, ...) also play a role in our choices.

Some features we value, that might raise the probability we consider a paper or project include the commitment and contribution to open science, the authors' engagement with our process, and the logic, communication, and transparent reasoning of the work. However, if a prominent research paper is within our scope and seems to have a strong potential for impact, we will prioritize it highly, whether or not it has these qualities.

2. Why does it need (more) evaluation, and what are some key issues and claims to vet?

We ask the people who suggest particular research, and experts in the field:

• What are (some of) the authors’ key/important claims that are worth evaluating?

• What aspects of the evidence, argumentation, methods, and interpretation, are you unsure about?

• What particular data, code, proofs, and arguments would you like to see vetted? If it has already been peer-reviewed in some way, why do you think more review is needed?

Ultimate goals: what are we trying to optimize?

Prioritizing research to achieve these goals

As we weigh research to prioritize for evaluation, we need to balance directly having a positive impact against building our ability to have an impact in the future.

A. Direct impact (‘score goals now’)

Importance

What is the direct impact potential of the research?

This is a massive question many have tried to address (see sketches and links below). We respond to uncertainty around this question in several ways, including:

• Consulting a range of sources, not only EA-linked sources.
• Scoping what other sorts of work are representative inputs to GP-relevant work.

  • Get a selection of seminal GP publications; look back to see what they are citing and categorize by journal/field/keywords/etc.

Neglectedness

Where is the current journal system failing GP-relevant work the most . . . in ways we can address?

Tractability

1. “Evaluability” of research: Where does the UJ approach yield the most insight or value of information?

2. Existing expertise: Where do we have field expertise on the UJ team? This will help us commission stronger evaluations.

3. "Feedback loops": Could this research influence concrete intervention choices? Does it predict near-term outcomes? If so, observing these choices and outcomes and getting feedback on the research and our evaluation can yield strong benefits.

Consideration/discussion: How much should we include research with indirect impact potential (theoretical, methodological, etc.)?

B. Sustainability: funding, support, participation

Moreover, we need to consider how the research evaluation might support the sustainability of The Unjournal and the broader general project of open evaluation. We may need to strike a balance between work informing the priorities of various audences, including:

• Relevance to stakeholders and potential supporters

• Clear connections to impact; measurability

• Support from relevant academic communities

• Support from open science

Consideration/discussion: What will drive further interest and funding?

C. Credibility, visibility, driving positive institutional change

Finally, we consider how our choices will increase the visibility and solidify the credibility of The Unjournal and open evaluations. We consider how our work may help drive positive institutional change. We aim to:

• Interest and involve academics—and build the status of the project.

• Commission evaluations that will be visibly useful and credible.

• ‘Benchmark traditional publication outcomes’, track our predictiveness and impact.

• Have strong leverage over research "outcomes and rewards."

• Increase public visibility and raise public interest.

• Bring in supporters and participants.

• Achieve substantial output in a reasonable time frame and with reasonable expense.

• Maintain goodwill and a justified reputation for being fair and impartial.

We hope we have identified the important considerations (above); but we may be missing key points. We continue to engage discussion and seek feedback, to hone and improve our processes and approaches.

Data: what are we evaluating/considering?

An important part of making this a success will be to spread the word, to get positive attention for this project, to get important players on board, network externalities, and change the equilibrium. We are also looking for specific feedback and suggestions from "mainstream academics" in Economics, Psychology, and policy/program evaluation, as well as from the Open Science and EA communities.

Key points to convey

See

As social media blurbs

How and where to promote and share

15 Dec 2023: Our main current process involves

• Submitted and (internally/externally) suggested research

• Prioritization ratings and discussion by Unjournal field specialists

• Feedback from field specialist area teams

• A final decision by the management team, guided by the above

See (also embedded below) for more details of the proposed process.

Evaluation guidelines and criteria

For more information about what we are asking evaluators to do, see:

• For Prospective Evaluators

• Guidelines for Evaluators

Choosing and working with evaluators

How do we choose evaluators?

• We follow standard procedures, considering complementary expertise, interest, and cross-citations, as well as checking for conflicts of interest. (See our internal guidelines for choosing evaluators.)

• We aim to consult those who have opted-in to our evaluator pool first.

• We favor evaluators with a track record of careful, in-depth, and insightful evaluation — while giving ECRs a chance to build such a record.

Why do we pay evaluators?

For several reasons... (for more discussion, see Why pay evaluators (reviewers)?)

• It's equitable, especially for those not getting "service credit" for their refereeing work from their employer.

• Paying evaluators can reduce adverse selection and conflicts of interest —arguably inherent to the traditional process where reviewers work for free.

• We need to use explicit incentives while The Unjournal grows.

• We can use payment as an incentive for high-quality work, and to access a wider range of expertise, including people not interested in submitting their own work to The Unjournal.

Evaluator concerns

Can I submit an evaluation anonymously? How will you protect my anonymity?

Yes, we allow evaluators to choose whether they wish to remain anonymous or "sign" their evaluations. See Protecting anonymity.

I'm concerned about making my evaluation public; what if I make a mistake or write something I later regret?

To limit this concern:

1. You can choose to make your evaluation anonymous. You can make this decision from the outset (this is preferable) or later, after you've completed your review.

2. Your evaluation will be shared with the authors before it is posted, and they will be given two weeks to respond before we post. If they cite what they believe are any major misstatements in your evaluation, we will give you the chance to correct these.

3. It is well-known that referee reports and evaluations are subject to mistakes. We expect most people who read your evaluation will take this into account.

4. You can add an addendum or revision to your evaluation later on (see below).

Can I redact my evaluation after it's published through The Unjournal?

We will put your evaluation on PubPub and give it a DOI. It cannot be redacted in the sense that this initial version will remain on the internet in some format. But you can add an addendum to the document later, which we will post and link, and the DOI can be adjusted to point to the revised version.

Are the research authors involved in The Unjournal's review process and do they give consent?

See the For research authors FAQ as well as the "Direct evaluation" track.

We have two main ways that papers and research projects enter the Unjournal process:

1. Authors submit their work; if we believe the work is relevant, we assign evaluators, and so on.

2. We select research that seems potentially influential, impactful, and relevant for evaluation. In some cases, we request the authors' permission before sending out the papers for evaluation. In other cases (such as where senior authors release papers in the prestigious NBER and CEPR series) we contact the authors and request their engagement before proceeding, but we don't ask for permission.

For either track, authors are invited to be involved in several ways:

• Authors are informed of the process and given an opportunity to identify particular concerns, request an embargo, etc.

• Evaluators can be put in touch with authors (anonymously) for clarification questions.

• Authors are given a two-week window to respond to the evaluations (this response is published as well) before the evaluations are made public. They can also respond after the evaluations are released.

Can I share this evaluation? What else can I do with it?

If you are writing a signed evaluation, you can share it or link it on your own pages. Please wait to do this until after we have given the author a chance to respond and posted the package.

Otherwise, if you are remaining anonymous, please do not disclose your connection to this report.

Going forward:

• We may later invite you to write and evaluate more about this piece of research . . .

• . . . and to help us judge prizes (e.g., the Impactful Research Prize (pilot)).

• We may ask if you want to be involved in replication exercises (e.g., through the Institute for Replication).

• As a general principle, we hope and intend always to see that you are fairly compensated for your time and effort.

Evaluation value

What value do these evaluations provide, and for whom?

The evaluations provide at least three types of value, helping advance several paths in our theory of change:

1. For readers and users: Unjournal evaluations assess the reliability and usefulness of the paper along several dimensions—and make this public, so other researchers and policymakers can learn from them.

2. For careers and improving research: Evaluations provide metrics of quality. In the medium term, these should provide increased and accelerated career value, improving the research process. We aim to build metrics that are credibly comparable to the current "tier" of journal a paper is published in. But we aim to do this better in several ways:

   • More quickly, more reliably, more transparently, and without the unproductive overhead of dealing with journals (see 'reshaping evaluation')

   • Allowing flexible, transparent formats (such as dynamic documents), thus improving the research process, benefiting research careers, and hopefully improving the research itself in impactful areas.

3. Feedback and suggestions for authors: We expect that evaluators will provide feedback that is relevant to the authors, to help them make the paper better.

See "what https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/faq-interaction/referees-evaluators#what-value-do-these-evaluations-provide-and-for-whom

Evaluation quality and misunderstandings

What should I prioritize in my evaluation process?

See our guidelines for evaluators.

'This paper is great, I would accept it without changes, what should I write/do?"

We still want your evaluation and ratings. Some things to consider as an evaluator in this situation.

A paper/project is not only a good to be judged on a single scale. How useful is it, and to who or what? We'd like you discuss its value in relation to previous work, it’s implications, what it suggests for research and practice, etc.

Even if the paper is great...

1. Would you accept it in the “top journal in economics”? If not, why not?

2. Would you hire someone based on this paper?

3. Would you fund a major intervention (as a government policymaker, major philanthropist, etc.) based on this paper alone? If not, why not

4. What are the most important and informative results of the paper?

5. Can you quantify your confidence in these 'crucial' results, and their replicability and generalizability to other settings? Can you state your probabilistic bounds (confidence or credible intervals) on the quantitative results (e.g., 80% bounds on QALYs/DALYs/or WELLBYs per $1000)

6. Would any other robustness checks or further work have the potential to increase your confidence (narrow your belief bounds) in this result? Which?

7. Do the authors make it easy to reproduce the statistical (or other) results of the paper from shared data? Could they do more in this respect?

8. Communication: Did you understand all of the paper? Was it easy to read? Are there any parts that could have been better explained?

9. Is it communicated in a way that would it be useful to policymakers? To other researchers in this field, or in the general discipline?

Paths to suggest research

Research can be "submitted" by authors (here) or "suggested" by others. For a walk-through on suggesting research, see this video example.

There are two main paths for making suggestions: through our survey form or through Airtable.

1. Through our survey form

Anyone can suggest research using the survey form at https://bit.ly/ujsuggestr. (Note, if you want to "submit your own research," go to bit.ly/ujsubmitr.) Please include the following steps:

Review The Unjournal's Guidelines

Begin by reviewing The Unjournal's guidelines on What research to target to get a sense of the research we cover and our priorities. Look for high-quality research that 1) falls within our focus areas and 2) would benefit from (further) evaluation.

When in doubt, we encourage you to suggest the research anyway.

Fill out the Suggestion Form

Navigate to The Unjournal's Suggest Research Survey Form. Most of the fields here are optional. The fields ask the following information:

• Who you are: Let us know who is making the suggestion (you can also choose to stay anonymous).

  • If you leave your contact information, you will be eligible for financial "bounties" for strong suggestions.

  • If you are already a member of The Unjournal's team, additional fields will appear for you to link your suggestion to your profile in the Unjournal's database.

• Research Label: Provide a short, descriptive label for the research you are suggesting. This helps The Unjournal quickly identify the topic at a glance.

• Research Importance: Explain why the research is important, its potential impact, and any specific areas that require thorough evaluation.

• Research Link: Include a direct URL to the research paper. The Unjournal prefers research that is publicly hosted, such as in a working paper archive or on a personal website.

• Peer Review Status: Inform about the peer review status of the research, whether it's unpublished, published without clear peer review, or published in a peer-reviewed journal.

• "Rate the relevance": This represents your best-guess at how relevant this work is for The Unjournal to evaluate, as a percentile relative to other work we are considering.

• Research Classification: Choose categories that best describe the research. This helps The Unjournal sort and prioritize suggestions.

• Field of Interest: Select the outcome or field of interest that the research addresses, such as global health in low-income countries.

Complete all the required fields and submit your suggestion. The Unjournal team will review your submission and consider it for future evaluation. You can reach out to us at contact@unjournal.org with any questions or concerns.

2. For Field Specialists and managers: via Airtable

People on our team may find it more useful to suggest research to The Unjournal directl via the Airtable. See this document for a guide to this. (Please request document permission to access this explanation.)

Further guidance

Aside on setting the prioritization ratings: In making your subjective prioritization rating, please consider “What percentile do you think this paper (or project) is relative to the others in our database, in terms of ‘relevance for The UJ to evaluate’?” (Note this is a redefinition; we previously considered these as probabilities.) We roughly plan to commission the evaluation of about 1 in 5 papers in the database, the ‘top 20%’ according to these percentiles. Please don’t consider the “publication status or the “author's propensity to engage” in this rating. We will consider those as separate criteria.

Notes for field specialists/Unjournal Team

Please don’t enter only the papers you think are ‘very relevant’; please enter in all research that you have spent any substantial time considering (more than a couple minutes). If we all do this, we should all aim for our percentile ratings to be approximately normally distributed; evenly spread over the 1-100% range.

You can request a conditional embargo by emailing us at contact@unjournal.org, or via the submission/response form. Please explain what sort of embargo you are asking for, and why. By default, we'd like Unjournal evaluations to be made public promptly.

However, we may make exceptions in special circumstances particularly

• for very early-career researchers who are not clearly #high-professional-status-less-career-sensitive,

• where the research is not obviously already influencing a substantial amount of funding in impact-relevant areas, or substantially influencing policy considerations

If there is an early-career researcher on the authorship team, we may allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite.

For example, we might grant an embargo that lasts until after a PhD/postdoc’s upcoming job market or until after publication in a mainstream journal, with a hard maximum of 14 months. (Of course, embargoes can be ended early at the request of the authors.)

In exceptional circumstances we may consider granting a "conditional indefinite embargo."

Some examples of possible embargos (need approval)

Note: the above are all exceptions to our regular rules, examples of embargos we might or might not agree to.

David Reinstein, Nov 2024: Over the last six months we have considered and evaluated a small amount of work under this “Applied & Policy Stream”. We are planning to continue this stream for the forseeable future.

Why have an “Applied & Policy Stream”?

Much of the most impactful research is not aimed at academic audiences and may never be submitted to academic journals. It is written in formats that are very different from traditional academic outputs, and cannot be easily judged by academics using the same standards. Nonetheless, this work may use technical approaches developed in academia, making it important to gain expert feedback and evaluation.

The Unjournal can help here. However, to avoid confusion, we want to make this clearly distinct from our main agenda, which focuses on impactful academically-aimed research.

Our “Applied & Policy Stream” will be clearly labeled as separate from our main stream. This may constitute roughly 10 or 15% of the work that we cover. Below, we refer to this as the “applied stream” for brevity.

What should be included in the Policy stream?

Our considerations for prioritizing this work are generally the same as for our academic stream – is it in the fields that we are focused on, using approaches that enable meaningful evaluation and rating? Is it already having impact (e.g., influencing grant funding in globally-important areas)? Does it have the potential for impact, and if so, is it high-quality enough that we should consider boosting its signal?

We will particularly prioritize policy and applied work that uses technical methods that need evaluation by research experts, often academics.

This could include a range of applied research from EA/GP/LT linked organizations such as GPI, Rethink Priorities, Open Philanthropy, FLI, HLI, Faunalytics, etc., as well as EA-adjacent organizations and relevant government white papers.

How should our (evaluation etc.) policies differ here?

Ratings/metrics: As in the academic stream, this work will be evaluated for its credibility, usefulness, communication/logic, etc. However, we are not seeking to have this work assessed by the standards of academia in a way that yields a comparison to traditional journal tiers. Evaluators: Please ignore these parts of our interface; if you are unsure if it is relevant feel free to ask.

Evaluator selection, number, pay: Generally we want to continue to select academic research experts or non-academic researchers with strong academic and methodological background to do these evaluations. A key purpose of this applied stream is largely to bring research expertise, particularly from academia, to work that is not normally scrutinized by such experts.

The compensation may be flexible as well; in some cases the work may be more involved than for the academic stream and in some cases less involved. As a starting point we will begin by offering the same compensation as for the academic stream.

Careful flagging and signposting: To preserve the reputation of our academic-stream evaluations we need to make it clear, wherever people might see this work, that it is not being evaluated by the same standards as the academic stream and doesn't “count” towards those metrics.

Authors' permission in the Applied Stream

This research is more likely to fall into the category of #direct-evaluation-of-impactful-work, "already influencing a substantial amount of funding in impact-relevant areas, or substantially influencing policy considerations".

If the research itself is being funded by a global-impact focused foundation or donor, this will also constitute a strong prima facie reason to commission an evaluation (without requiring the authors' consent). See this post on the EA Forum.

1. We are targeting global priorities-relevant research...

2. With the potential for impact, and with the potential for Unjournal evaluations to have an impact (see our high-level considerations and our prioritization ratings discussions).

3. Our initial focus is quantitative work that informs global priorities (see linked discussion), especially in economics, policy, and social science, informing our Theory of Change.

4. We give a data presentation of the work we have already covered and the work we are prioritizing here, which will be continually updated.

But what does this mean in practice? What specific research fields, topics, and approaches are we likely to classify as 'relevant to evaluate'?

We give some lists and annotated examples below.

Fields, methods, and approaches

As of January 2024 The Unjournal focuses on ...

1. Research where the fundamental question being investigated involves human behavior and beliefs and the consequences of these. This may involve markets, production processes, economic constraints, social interactions, technology, the 'market of ideas', individual psychology, government processes, and more. However, the main research question should not revolve around issues outside of human behavior, such as physical science, biology, or computer science and engineering. These areas are out of our scope (at least for now).

2. Research that is fundamentally quantitative and uses scientific methods. It will generally involve or consider measurable inputs, choices, and outcomes; specific categorical or quantitative questions; analytical and mathematical reasoning; hypothesis testing and/or belief updating, etc.

3. Research that targets and addresses a single specific question or goals, or a small cluster. It should not mainly be a broad discussion and overview of other research or conceptual issues.

This to generally involves the academic fields:

• Economics

• Applied Statistics (and some other applied math)

• Psychology

• Political Science

• Other quantitative social science fields (perhaps Sociology)

• Applied "business school" fields: finance, accounting, operations, etc.

• Applied "policy and impact evaluation" fields

• Life science/medicine where it targets human behavior/social science

These discipline/field boundaries are not strict; they may adapt as we grow

Why this field/method focus?

These were chosen in light of two main factors:

1. Our founder and our team is most comfortable assessing and managing the consideration of research in these areas.

2. These fields seem to be particularly amenable to, and able to benefit from our journal-independent evaluation approach. Other fields, such as biology, are already being 'served' by strong initiatives like Peer Communities In.

Ex.: work we included/excluded based on field/method

Outcomes, focus areas, and causes

The Unjournal's mission is to prioritize

• research with the strongest potential for a positive impact on global welfare

• where public evaluation of this research will have the greatest impact

Given this broad goal, we consider research into any cause, topic, or outcome, as long as the research involves fields, methods, and approaches within our domain (see above), and as long as the work meets our other requirements (e.g., research must be publicly shared without a paywall).

While we don't have rigid boundaries, we are nonetheless focusing on certain areas:

Fields

(As of Jan. 2024) we have mainly commissioned evaluations of work involving development economics and health-related outcomes and interventions in low-and middle-income countries.

As well as research involving

• Environmental economics, conservation, harm to human health

• The social impact of AI and emerging technologies

• Economics, welfare, and governance

• Catastrophic risks; predicting and responding to these risks

• The economics of innovation; scientific progress and meta-science

• The economics of health, happiness, and wellbeing

We are currently prioritizing further work involving

• Psychology, behavioral science, and attitudes: the spread of misinformation; other-regarding preferences and behavior; moral circles

• Animal welfare: markets, attitudes

• Methodological work informing high-impact research (e.g., methods for impact evaluation)

We are also considering prioritizing work involving

• AI governance and safety

• Quantitative political science (voting, lobbying, attitudes)

• Political risks (including authoritarian governments and war and conflict)

• Institutional decisionmaking and policymaking

• Long-term growth and trends; the long-term future of civilization; forecasting

Examples of work we chose to prioritize or de-prioritize based on focus area

As noted in Process: prioritizing research, we ask people who suggest research to provide a numerical 0-100 rating:

We also ask people within our team to act as 'assessors' to give as second and third opinions on this. This 'prioritization rating' is one of the criteria we will use to determine whether to commission research to be evaluated (along with author engagement, publication status, our capacity and expertise, etc.) Again, see the previous page for the current process.

So what goes into this "prioritization rating"; what does it mean?

We are working on a set of notes on this, fleshing this out and giving specific examples. At the moment this is available to members of our team only (ask for access to "Guidelines for prioritization ratings (internal)"). We aim to share a version of this publicly once it converges, and once we can get rid of arbitrary sensitive examples.

Some key points

I. This is not the evaluation itself. It is not an evaluation of the paper's merit per se:

• Influential work, and prestigious work in influential areas may be highly prioritized regardless of its rigor and quality

• The prioritization rating might consider quality for work that seems potentially impactful, which does not seem particularly prestigious or influential. Here aspects like writing clarity, methodological rigor, etc., might put it 'over the bar'. However, even here these will tend to be based on rapid and shallow assessments, and should not be seen as meaningful evaluations of research merit.

II. These ratings will be considered along with the discussion by the field team and the management. Thus is helpful if you give a justification and explanation for your stated rating.

One possible way of considering the rating criteria

Key attributes/factors

Define/consider the following ‘attributes’ of a piece of research:

1. Global decision-relevance/VOI: Is this research decision-relevant to high-value choices and considerations that are important for global priorities and global welfare?

2. Prestige/prominence: Is the research already prominent/valued (esp. in academia), highly cited, reported on, etc?

3. Influence: Is the work already influencing important real-world decisions and considerations?

Obviously, these are not binary factors; there is a continuum for each. But for the sake of illustration, consider the following flowcharts.

Prestigious work

"Fully baked": Sometimes prominent researchers release work (e.g., on NBER) that is not particularly rigorous or involved, which may have been put together quickly. This might be research that links to a conference they are presenting at, to their teaching, or to specific funding or consulting. It may be survey/summary work, perhaps meant for less technical audiences. The Unjournal tends not to prioritize such work, or at least not consider it in the same "prestigious" basket (although there will be exceptions). In the flowchart above, we contrast this with their "fully-baked" work.

Decision-relevant, prestigious work: Suppose the research is both ‘globally decision-relevant’ and prominent. Here, if the research is in our domain, we probably want to have it publicly evaluated. This is basically the case regardless of its apparent methodological strength. This is particularly true if it has been recently made public (as a working paper), if it has not yet been published in a highly-respected peer-reviewed journal, and if there are non-straightforward methodological issues involved.

Prestigious work that seems less globally-relevant: We generally will not prioritize this work unless it adds to our mission in other ways (see, e.g., our ‘sustainability’ and ‘credibility’ goals here). In particular we will prioritize such research more if:

• It is presented in innovative, transparent formats (e.g., dynamic documents/open notebooks, sharing code and data)

• The research indirectly supports more globally-relevant research, e.g., through…

  • Providing methodological tools that are relevant to that ‘higher-value’ work

  • Drawing attention to neglected high-priority research fields (e.g., animal welfare)

Less prestigious work

(If the flowchart below does not render, please refresh your browser; you may have to refresh twice.)

Decision-relevant, influential (but less prestigious) work: E.g., suppose this research might be cited by a major philanthropic organization as guiding its decision-making, but the researchers may not have strong academic credentials or a track record. Again, if this research is in our domain, we probably want to have it publicly evaluated. However, depending on the rigor of the work and the way it is written, we may want to explicitly class this in our ‘non-academic/policy’ stream.

Decision-relevant, less prestigious, less-influential work: What about for less-prominent work with fewer academic accolades that is not yet having an influence, but nonetheless seems to be globally decision-relevant? Here, our evaluations seem less likely to have an influence unless the work seems potentially strong, implying that our evaluations, rating, and feedback could boost potentially valuable neglected work. Here, our prioritization rating might focus more on our initial impressions of things like …

• Methodological strength (this is a big one!)

• Rigorous logic and communication

• Open science and robust approaches

• Engagement with real-world policy considerations

Again: the prioritization process is not meant to be an evaluation of the work in itself. It’s OK to do this in a fairly shallow way.

In future, we may want to put together a loose set of methodological ‘suggestive guidelines’ for work in different fields and areas, without being too rigid or prescriptive. (To do: we can draw from some existing frameworks for this [ref].)

The Unjournal commissions public expert evaluation of open-access research: to make rigorous research more impactful and impactful research more rigorous. We prioritize work that concretely informs global priorities, focusing on economics, quantitative social science, and policy.

Today's research evaluation process is out-of-date. Traditional journals capture rents while discouraging innovative formats. The current publish-or-perish incentive diverts researcher energy away from their core work and towards playing strategic journal-submission games.

The Unjournal provides open, rigorous evaluation, focusing on credibility, robustness, transparency, communication, and usefulness. We make it easier for researchers to get feedback and credible assessment of their work, so they can focus on doing better and more useful research. We publish and disseminate these evaluation reports and benchmarked ratings; often far earlier than traditional journals. This helps policymakers understand what research to trust and how to use it, and it helps other researchers and students learn from the critique and discussion.

Click on the cards below to find out more about our mission, organizational structure, and ways to collaborate, or 'ask or search our page' for answers to your questions.

Direct evaluation of prominent work

In addition to soliciting research submissions by authors, we have a process for sourcing and prioritizing unsubmitted research for evaluation. For some of this research we ask for author engagement but do not require their permission.

Direct evaluation of "prestige work": eligibility rules & guidelines

NBER

All NBER working papers are generally eligible, with rare exceptions.

CEPR

We treat these on a case-by-case basis and use discretion. All CEPR members are reasonably secure and successful, but their co-authors might not be, especially if these co-authors are PhD students they are supervising.

Journal-published papers (i.e., 'post-publication evaluation')

In some areas and fields (e.g., psychology, animal product markets) the publication process is relatively rapid or it may fail to engage general expertise. In general, all papers that are already published in peer-reviewed journals are eligible for our direct track.

Papers or projects posted in any other working paper (pre-print) series

These are eligible (without author permission) if all authors/all lead authors "have high professional status" or are otherwise less career-sensitive to the consequences of this evaluation.

High professional status/less career-sensitive

We define this (at least for economics) as:

• having a tenured or ‘long term’ positions at well-known, respected universities or other research institutions, or

• having a tenure-track positions at a top universities (e.g., top-20 globally by some credible ranking) and having published one or more papers in a "top-five-equivalent" journal, or

• clearly not pursuing an academic career (e.g., the "partner at the aid agency running the trial").

On the other hand, if one or more authors is a PhD student close to graduation or an untenured academic outside a "top global program,’’ then we will ask for permission and potentially offer an embargo.

Exceptions to these exceptions

If the PhD student or untenured academic is otherwise clearly extremely high-performing by conventional metrics; e.g., an REStud "tourist" or someone with multiple published papers in top-5 journals. In such cases the paper might be considered eligible for direct evaluation.

See also #direct-evaluation-of-impactful-work

Direct evaluation of "impactful work"

We will also evaluate work directly, without requiring author permission, where it is clear that this research is already influencing a substantial amount of funding in impact-relevant areas, or substantially influencing policy considerations. Much of this work will be evaluated as part of our "Applied and Policy" Track.

'Dynamic Documents' are projects or papers that are developed using prominent tools such as R-markdown or JuPyTer notebooks (the two most prominent tools).

The salient features and benefits of this approach include:

• Integrated data analysis and reporting means the data analysis (as well as math/simulations) is done and reported in the same space that the results and discussion are presented. This is made possible through the concealment of 'code blocks'.

• Transparent reporting means you can track exactly what is being reported and how it was constructed:

  • Making the process a lot less error-prone

  • Helping readers understand it better (see 'explorable explanations')

  • Helping replicators and future researchers build on it

• Other advantages of these formats (over PDFs for example) include:

  • Convenient ‘folding blocks’

  • Margin comments

  • Hoverable notes and links

  • Integrating interactive tools

Better examples, the case for dynamic documents

Reinstein's own examples

Some quick examples from my own work in progress (but other people have done it much better)

Other (randomly selected) examples

: An overview of what we are asking; payment and recognition details

: The Unjournal's evaluation guidelines, considering our priorities and criteria, the metrics we ask for, and how these are considered.

Other sections and subsections provide further resources, consider future initiatives, and discuss our rationales.

Why numerical ratings?

Ultimately, we're trying to replace the question of "what tier of journal did a paper get into?" with "how highly was the paper rated?" We believe this is a more valuable metric. It can be more fine-grained. It should be less prone to gaming. It aims to reduce randomness in the process, through things like 'the availability of journal space in a particular field'. See our discussion of .

To get to this point, we need to have academia and stakeholders see our evaluations as meaningful. We want the evaluations to begin to have some value that is measurable in the way “publication in the AER” is seen to have value.

While there are some ongoing efforts towards journal-independent evaluation, these . Typically, they either have simple tick-boxes (like "this paper used correct statistical methods: yes/no") or they enable descriptive evaluation without an overall rating. As we are not a journal, and we don’t accept or reject research, we need another way of assigning value. We are working to determine the best way of doing this through quantitative ratings. We hope to be able to benchmark our evaluations to "traditional" publication outcomes. Thus, we think it is important to ask for both an overall quality rating and a journal ranking tier prediction.

Why these categories?

In addition to the overall assessment, we think it will be valuable to have the papers rated according to several categories. This could be particularly helpful to practitioners who may care about some concerns more than others. It also can be useful to future researchers who might want to focus on reading papers with particular strengths. It could be useful in meta-analyses, as certain characteristics of papers could be weighed more heavily. We think the use of categories might also be useful to authors and evaluators themselves. It can help them get a sense of what we think research priorities should be, and thus help them consider an overall rating.

However, these ideas have been largely ad-hoc and based on the impressions of our management team (a particular set of mainly economists and psychologists). The process is still being developed. Any feedback you have is welcome. For example, are we overemphasizing certain aspects? Are we excluding some important categories?

We are also researching other frameworks, templates, and past practice; we hope to draw from validated, theoretically grounded projects such as .

Why ask for credible intervals?

In eliciting expert judgment, it is helpful to differentiate the level of confidence in predictions and recommendations. We want to know not only what you believe, but how strongly held your beliefs are. If you are less certain in one area, we should weigh the information you provide less heavily in updating our beliefs. This may also be particularly useful for practitioners. Obviously, there are challenges to any approach. Even experts in a quantitative field may struggle to convey their own uncertainty. They may also be inherently "poorly calibrated" (see discussions and tools for ). Some people may often be "confidently wrong." They might state very narrow "credible intervals", when the truth—where measurable—routinely falls outside these boundaries. People with greater discrimination may sometimes be underconfident. One would want to consider and As a side benefit, this may be interesting for research , particularly as The Unjournal grows. We see 'quantifying one's own uncertainty' as a good exercise for academics (and everyone) to engage in.

"Weightings" for each rating category (removed for now)

Adjustments to metrics and guidelines/previous presentations

Pre-October 2023 'ratings with weights' table, provided for reference (no longer in use)

We had included the note:

We give the previous weighting scheme in a fold below for reference, particularly for those reading evaluations done before October 2023.

As well as:

Elsewhere in that page we had noted:

Pre-2024 ratings and uncertainty elicitation, provided for reference (no longer in use)

[FROM PREVIOUS GUIDELINES:]

You may feel comfortable giving your "90% confidence interval," or you may prefer to give a "descriptive rating" of your confidence (from "extremely confident" to "not confident").

[Previous...] Remember, we would like you to give a 90% CI or a confidence rating (1–5 dots), but not both.

And, for the 'journal tier' scale:

Previous 'descriptions of ratings intervals'

[Previous guidelines]: The description folded below focuses on the "Overall Assessment." Please try to use a similar scale when evaluating the category metrics.