Evaluation guidelines and criteria

For more information about what we are asking evaluators to do, see:

Choosing and working with evaluators

How do we choose evaluators?

• We follow standard procedures, considering complementary expertise, interest, and cross-citations, as well as checking for conflicts of interest. (See our internal guidelines for .)

• We aim to consult those who have first.

• We favor evaluators with a track record of careful, in-depth, and insightful evaluation — while giving ECRs a chance to build such a record.

Why do we pay evaluators?

For several reasons... (for more discussion, see )

• It's equitable, especially for those not getting "service credit" for their refereeing work from their employer.

• Paying evaluators can reduce adverse selection and conflicts of interest —arguably inherent to the traditional process where reviewers work for free.

Evaluator concerns

Can I submit an evaluation anonymously? How will you protect my anonymity?

Yes, we allow evaluators to choose whether they wish to remain anonymous or "sign" their evaluations. See .

I'm concerned about making my evaluation public; what if I make a mistake or write something I later regret?

To limit this concern:

1. You can choose to make your evaluation anonymous. You can make this decision from the outset (this is preferable) or later, after you've completed your review.

2. Your evaluation will be shared with the authors before it is posted, and they will be given two weeks to respond before we post. If they cite what they believe are any major misstatements in your evaluation, we will give you the chance to correct these.

3. It is well-known that referee reports and evaluations are subject to mistakes. We expect most people who read your evaluation will take this into account.

Can I redact my evaluation after it's published through The Unjournal?

We will put your evaluation on and give it a DOI. It cannot be redacted in the sense that this initial version will remain on the internet in some format. But you can add an addendum to the document later, which we will post and link, and the DOI can be adjusted to point to the revised version.

Are the research authors involved in The Unjournal's review process and do they give consent?

See the FAQ as well as the .

We have two main ways that papers and research projects enter the Unjournal process:

1. Authors submit their work; if we believe the work is relevant, we assign evaluators, and so on.

2. We select research that seems potentially influential, impactful, and relevant for evaluation. In some cases, we request the authors' permission before sending out the papers for evaluation. In other cases (such as where senior authors release papers in the prestigious

For either track, authors are invited to be involved in several ways:

• Authors are informed of the process and given an opportunity to identify particular concerns, request an embargo, etc.

• Evaluators can be put in touch with authors (anonymously) for clarification questions.

• Authors are given a two-week window to respond to the evaluations (this response is published as well) before the evaluations are made public. They can also respond after the evaluations are released.

Can I share this evaluation? What else can I do with it?

If you are writing a signed evaluation, you can share it or link it on your own pages. Please wait to do this until after we have given the author a chance to respond and posted the package.

Otherwise, if you are remaining anonymous, please do not disclose your connection to this report.

Going forward:

• We may later invite you to write and evaluate more about this piece of research . . .

• . . . and to help us judge prizes (e.g., the ).

• We may ask if you want to be involved in replication exercises (e.g., through the

Evaluation value

What value do these evaluations provide, and for whom?

The evaluations provide at least three types of value, helping advance several paths in our :

1. For readers and users: Unjournal evaluations assess the reliability and usefulness of the paper along several dimensions—and make this public, so other researchers and policymakers can learn from them.

2. For careers and improving research: Evaluations provide metrics of quality. In the medium term, these should provide increased and accelerated career value, improving the research process. We aim to build metrics that are credibly comparable to the current "tier" of journal a paper is published in. But we aim to do this better in several ways:

See "what

Evaluation quality and misunderstandings

What should I prioritize in my evaluation process?

See our .

'This paper is great, I would accept it without changes, what should I write/do?"

We still want your evaluation and ratings. Some things to consider as an evaluator in this situation.

A paper/project is not only a good to be judged on a single scale. How useful is it, and to who or what? We'd like you discuss its value in relation to previous work, it’s implications, what it suggests for research and practice, etc.

Even if the paper is great...

1. Would you accept it in the “top journal in economics”? If not, why not?

2. Would you hire someone based on this paper?

3. Would you fund a major intervention (as a government policymaker, major philanthropist, etc.) based on this paper alone? If not, why not

: An overview of what we are asking; payment and recognition details

: The Unjournal's evaluation guidelines, considering our priorities and criteria, the metrics we ask for, and how these are considered.

Other sections and subsections provide further resources, consider future initiatives, and discuss our rationales.

Our status

Submitting for payments and expenses

Evaluators: To claim your payment for evaluation work, please complete .

You will receive your payment via a transfer (they may ask you for your bank information if you don't have an account with them).

We aim to process all payments within one week.

Confidentiality: Your identity will only be visible to The Unjournal administrators for the purposes of making this payment. If you have concerns about the payment process, please contact us at .

Team members and contractors: Please refer to the internal Coda workspace for expense submission procedures.

is making research better by evaluating what really matters. We aim to make rigorous research more impactful and impactful research more rigorous.

The academic journal system is out-of-date. It wastes resources, discourages innovation, and encourages rent-seeking.

The Unjournal is not a journal. We don’t publish research. Instead, we commission (and pay for) open, rigorous expert evaluation of publicly-hosted research. We make it easier for researchers to get feedback and credible ratings of their work, so they can focus on doing better research rather than journal-shopping.

We currently focus on quantitative work that informs global priorities, especially in economics, policy, and social science. Our evaluation packages help policymakers and practitioners understand which research to trust and how to use it. These provide open critique and discussion to benefit students and the research community.

Click on the cards below to find out more about our mission, organizational structure, and ways to collaborate, or 'ask or search our page' for answers to your questions.

Current status (January 2026)

Evaluator prizes (2024-25)

For the 2024-25 cycle, we are focusing on evaluator recognition, with $6,500 in prizes for evaluators who provided exceptionally credible, insightful, and detailed evaluations. We are currently finalizing the winners and will announce them soon.

Author recognition (2024-25)

Given limited funds and a relatively small pool of author-submitted work during 2024-25, we are shifting from monetary "author prizes" to recognition-based awards highlighting two dimensions:

• Flowing Water Scholar: Recognizing the most valuable author engagement and response—openness, epistemic modesty, willingness to update, growth mindset, and constructive dialogue with evaluators.

• Polaris Research: Recognizing the strongest and most useful research, based on highly credible and detailed evaluation manager reports, careful evaluations, and high "merited journal tier" ratings.

Future prizes (2026 and beyond)

We aim to establish an ongoing author prize with a monetary component for 2026 and beyond. This will particularly reward authors who submit their work and engage meaningfully with the evaluation process. The structure and scale will depend on our funding situation.

To be considered for future recognition:

• for Unjournal evaluation

• Engage with the evaluation process by responding substantively to evaluator feedback

• Sign up as an

Updates will be announced on and our . Contact with questions.

2023 Pilot Prize (completed)

Our inaugural Impactful Research Prize was awarded in early 2024. See for the full announcement.

Winners:

• First Prize ($2,500): Takahiro Kubo and co-authors for ""

• Second Prize ($1,000): Johannes Haushofer and co-authors for ""

We also recognized exceptional evaluators: Phil Trammell, Hannah Metzler, Alex Bates, and Robert Kubinec.

Note: In a subsection below, , we outline the basic requirements for submissions to The Unjournal.

Some key sections and subsections

Learn more about The Unjournal, our goals and policies

For authors, evaluators, and others interested in collaboration.

Write-ups of the main points for a few different audiences

Important benefits of journal-independent public evaluation and The Unjournal's approach, with links to deeper commentary

Description of our evaluation process

Other resources and reading

Groups we work with; comparing approaches

What research are we talking about? What will we cover?

Detail, progress, and internal planning

These are of more interest to people within our team; we are sharing these in the spirit of transparency.

A "best feasible plan" for going forward

Successful proposals (ACX, SFF), other applications, initiatives

Key resources and links for managers, advisory board members, staff, team members and others involved with The Unjournal project.

Note: we have moved some of this "internal interest content" over to our Coda.io knowledge base.

In a nutshell

In brief

(see our In a nutshell) wants your involvement, help, and feedback. We offer recognize and compensate people for their time and effort.

1. Read our evaluation packages here, add comments, cite the evaluations in your work, let us know how you found them useful and how we can improve.

2. Make Unjournal evaluations part of your process, using them to consider your funding and policy choices, encouraging your team or department to submit work for public evaluation, and make this part of your hiring, promotion, and research funding criteria.

3. Submit your research for Unjournal to evaluate , or by contacting . Suggest research for us to assess using .

4. Join our team: Complete (about 3–5 min) to apply for our...

   1. Evaluator pool: to be eligible to be commissioned and paid to evaluate and rate research, mainly in quantitative social science and policy
5. Do an Independent Evaluation to build your portfolio, receive guidance, and be eligible for promotion and prizes. See details

6. Suggest for us to focus on

Give us feedback: What did you find useful and how did you use the work? Is anything unclear? What could be improved? Email contact@unjournal.org, and we will respond. We will offer rewards for the most useful suggestions.

Overview and call

is the founder and co-director of The Unjournal. The organization is currently looking for field specialists and evaluators, as well as suggestions for relevant work for The Unjournal to evaluate.

The Unjournal is building a system for credible, public, journal-independent feedback and evaluation of research.

We maintain an open call for participants for several different roles:

1. (involving honorariums for time spent)

2. members (no time commitment)

3. (who will often also be on the Advisory Board)

You can express your interest (and enter our database) .

Evaluators

We will reach out to evaluators (a.k.a. "reviewers") on a case-by-case basis, appropriate for each paper or project being assessed. This is dependent on expertise, the researcher's interest, and a lack of conflict of interest.

Time commitment: Case-by-case basis. For each evaluation, for the amount of time to spend.

Compensation: We pay evaluators for their time and effort, targeting an average of $350 per evaluation including prizes and incentives (and aim to return to $450+ with future funding). .

Who we are looking for: We are putting together a list of people interested in being an evaluator and doing paid referee work for The Unjournal. We generally prioritize the pool of evaluators who signed up for our database before reaching out more widely.

Interested? Please fill out (about 3–5 min, same form for all roles or involvement).

Projects and papers

We are looking for high-quality, globally pivotal research projects to evaluate, particularly those embodying open science practices and innovative formats. We are putting out a call for relevant research. Please suggest research . (We offer bounties and prizes for useful suggestions – see note.) For details of what we are looking for, and some potential examples, and accompanying links.

You can also put .

Contact us

If you are interested in discussing any of the above in person, please email us () to arrange a conversation.

Note: This is under continual refinement; see our for more details.

Express your interest

To indicate your potential interest in roles at The Unjournal, please fill out and link (or upload) your CV or webpage.

• If you already filled out this form for a role that has changed titles, don't worry. You will still be considered for relevant and related roles in the future.

• If you add your name to this form, we may contact you to offer you the opportunity to do paid project work.

If you are interested in conducting paid research evaluation for The Unjournal, or in joining our advisory board, please complete the form linked .

Feel free to contact contact@unjournal.org with any questions.

Other ways to get involved

Not looking for a job but want to participate? See our page for other opportunities including:

• Submitting or suggesting research for evaluation

• Becoming an evaluator

• Joining as a field specialist

Additional information

The Unjournal, a US-registered 501(c)(3) nonprofit, is an equal-opportunity employer and contractor. We are committed to creating an inclusive environment for all employees, volunteers, and contractors. We do not discriminate on the basis of race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetic information, disability, age, or veteran status.

See our data protection statement .

In addition to the "work roles," we are looking to engage researchers, research users, meta-scientists, and people with experience in open science, open access, and management of initiatives similar to The Unjournal.

We are continually looking to enrich our general team and board, including our , These roles come with some compensation and incentives.

(Please see links and consider submitting an expression of interest).

Current focus

The Unjournal's pilot phase was completed in August 2023. We are now focused on:

1. Scaling evaluation coverage — Expanding the volume and range of research we evaluate

2. Building field specialist teams — Recruiting experts to monitor and prioritize research in key areas

3. Improving evaluation processes — Refining our metrics, guidelines, and workflow

See the vision and broad plan presented :

Key resources

I (David Reinstein) am an economist who left UK academia after 15 years to pursue a range of projects (). One of these is :

The Unjournal (with funding from the and the Survival and Flourishing Fund) organizes and funds public-journal-independent feedback and evaluation, paying reviewers for their work. We focus on research that is highly relevant to global priorities, especially in economics, social science, and impact evaluation. We encourage better research by making it easier for researchers to get feedback and credible ratings on their work.

Management committee members

Activities of those on the management committee may involve a combination of the following (although you can choose your focus):

• Contributing to the decision-making process regarding research focus, reviewer assignment, and prize distribution.

• Collaborating with other committee members on the establishment of rules and guidelines, such as determining the metrics for research evaluation and defining the mode of assessment publication.

• Helping plan The Unjournal’s future path.

Time commitment: A minimum of 15–20 hours per year.

Compensation: We have funding for a $57.50 per hour honorarium for the first 20 hours, with possible compensation for additional work. will be further compensated (at roughly $300–$450 per paper).

Who we are looking for: All applicants are welcome. We are especially interested in those involved in global priorities research (and related fields), policy research and practice, open science and meta-science, bibliometrics and scholarly publishing, and any other academic research. We want individuals with a solid interest in The Unjournal project and its goals, and the ability to meet the minimal time commitment. Applying is extremely quick, and those not chosen will be considered for other roles and work going forward.

Advisory board (AB) members

Beyond direct roles within The Unjournal, we're building a larger, more passive advisory board to be part of our network, to offer occasional feedback and guidance. There is essentially no minimum time commitment for advisory board members—only opportunities to engage. We sketch some of the expectations in the fold below.

Field Specialists and Unjournal Research Affiliates

Goals: Support open evaluation and impactful, credible research, helping The Unjournal keep on top of the latest, most relevant, influential, and globally-consequential work. Help raise awareness of The Unjournal, fostering a transition away from traditional journal peer review towards a more transparent, efficient, and rigorous approach.

What Field Specialists (FS's) and Unjournal Research Affiliates (URAs) do, specifically

Source, prioritize, and suggest research

Focus on a particular area of research, policy, or impactful outcome, in collaboration with other members of our area-focused teams.

Keep track of new or under-considered research with potential for impact and explain and assess the extent to which The Unjournal can add value by commissioning its evaluation. Provide a brief explanation of why each paper matters and how it aligns with The Unjournal’s approach.

Scope: Perhaps three papers per academic term; FS are encouraged to suggest more. This comes with compensation and incentives

Help track and curate this list of prioritized research, considering and discussing specific areas and issues that merit evaluation. We maintain a public database of research of potential interest .

Prioritize research submitted by authors or suggested by other team members

Write and share "second opinion" assessments of this research, considering the importance and relevance for our evaluation. Vote on a short list of papers in your area to determine whether we should commission these for evaluation. Attend (very occassional) 'field group meetings' to discuss this.

Evaluation management (FS)

Field Specialists may serve as 'evaluation managers' (with additional compensation). Evaluation managers help us identify and commission evaluators for work we have prioritized, help manage this evaluation process, and write a summary report on the evaluations

Represent The Unjournal

Understand our mission and be ready to explain it to colleagues and (optionally) on social media.

Benefits of being a Field Specialist or URA

• Public Leadership: It publicly signals your commitment to open science principles, innovation, and research impact.

• Professional Credibility & Recognition: URAs are chosen through a selective process, and will be publicly acknowledged. URAs gain valuable experience, strengthen their CV, and position themselves for potential future opportunities with The Unjournal (e.g., Field Specialist or management roles).

• Networking & Community: Build contacts with others in your field and with related interests. Stay on top of cutting-edge research, discuss and understand its value and connection to the field and to policy/impact.

Field Specialists: specifics and norms

See the page.

Time commitment: There is no specific time obligation—only opportunities to engage. We may also consult you occasionally on your areas of expertise. Perhaps 1–4 hours a month is a reasonable starting expectation for people already involved in doing or using research, plus potential additional paid assignments. Our document also provides some guidance on the nature of work and the time involved.

Compensation: We aim to fairly compensate people for time spent on work done to support The Unjournal, and to provide incentives for suggesting and helping to prioritize research for evaluation. See the page. will be compensated at roughly $300–$450 per project.

Who we are looking for: For the FS roles, we are seeking active researchers, practitioners, and stakeholders with a strong publication record and/or involvement in the research and/or research-linked policy and prioritization processes. For the AB, also people with connections to academic, governmental, or relevant non-profit institutions, and/or involvement in open science, publication, and research evaluation processes. People who can offer relevant advice, experience, guidance, or help communicate our goals, processes, and progress.

Interested? (about 3–5 min, using the same form for all roles).

Unjournal Research Affiliates: specifics and norms

Application criteria: Applicants will selected based on (1) their ability to assess, discuss, and communicate the global impact of research, (2) their understanding of The Unjournal approach, (3) their research skills, interests, and experience.

Low Time Commitment: This role is designed to fit into a busy academic or research schedule. You’ll stay involved at a comfortable level without overextending your commitments.

Compensation This role comes with no baseline compensation, but URAs are eligible for incentive compensation for suggesting papers, for doing high-value assessment for prioritization, and more (see ).

Expectations for URAs

1. Recommend 2–3 papers each academic term that you think have the potential to make a substantial impact. Provide a brief explanation of why each paper matters and how it aligns with The Unjournal’s approach. Earn compensation for your recommendations.

2. Provide Second Opinions & Prioritization: Complete three “second opinion” assessments each term, offering a concise evaluation of a paper’s relevance and importance. Contribute to broader discussions about research prioritization.

3. Vote & Engage: Participate in monthly voting on a short list of papers. Stay semi-active on Slack and Coda (with no strict requirement to check in regularly, beyond voting).

for this and other roles.

Self-contained public release:

Contact us

If you are interested in discussing any of the above in person, please email us () to arrange a conversation.

We invite you to (the same as that linked above) to leave your contact information and outline which parts of the project interest you.

Note: These descriptions are under continual refinement; see our for more details.

These are principally not research roles, but familiarity with research and research environments will be helpful, and there is room for research involvement depending on the candidate's interest, background, and skills/aptitudes.

TLDR: See In a nutshell

Podcasts, presentations, and video

Journal independent evaluation and The Unjournal

EA Anywhere (Youtube) – bridging the gap between EA and academia

• See slide deck (Link: ; offers comment access)

ReproducibiliTea podcast

Slide decks

Presentation for EA Anywhere, online event, 5 Nov. 2023 1-2pm ET

(Link: ; offers comment access)

"Slaying the journals": Google doc

Earlier discussion document, aimed at EA/global priorities, academic, and open-science audiences

"

• 2021 A shorter outline posted on

EA forum posts

The potential roles discussed below combine research-linked work with operations and administrative responsibilities. Overall, this may include some combination of:

Note on other versions of this content.

Initiative: ‘independent evaluations’

is seeking academics, researchers, and students to submit structured evaluations of the most impactful research emerging in the social sciences. Strong evaluations will be posted or linked on our , offering readers a perspective on the implications, strengths, and limitations of the research. These evaluations can be submitted using for academic-targeted research or for more applied work; evaluators can publish their name or maintain anonymity; we also welcome collaborative evaluation work. We will facilitate, promote, and encourage these evaluations in several ways, described below.

Who should do these evaluations?

We are particularly looking for people with research training, experience, and expertise in quantitative social science and statistics including cost-benefit modeling and impact evaluation. This could include professors, other academic faculty, postdocs, researchers outside of academia, quantitative consultants and modelers, PhD students, and students aiming towards PhD-level work (pre-docs, research MSc students etc.) But anyone is welcome to give this a try — when in doubt, piease go for it.

We are also happy to support collaborations and group evaluations. There is a good track record for this — see: “”, ASAPBio’s, and for examples in this vein. We may also host live events and/or facilitate asynchronous collaboration on evaluations

Instructors/PhD, MRes, Predoc programs: We are also keen to work with students and professors to integrate ‘independent evaluation assignments’ (aka ‘learn to do peer reviews’) into research training.

Why should you do an evaluation?

Your work will support The Unjournal’s core mission — improving impactful research through journal-independent public evaluation. In addition, you’ll help research users (policymakers, funders, NGOs, fellow researchers) by providing high quality detailed evaluations that rate and discuss the strengths, limitations, and implications of research.

Doing an independent evaluation can also help you. We aim to provide feedback to help you become a better researcher and reviewer. We’ll also give prizes for the strongest evaluations. Lastly, writing evaluations will help you build a portfolio with The Unjournal, making it more likely we will commission you for paid evaluation work in the future.

Which research?

We focus on rigorous, globally-impactful research in quantitative social science and policy-relevant research. (See for details.) We’re especially eager to receive independent evaluations of:

1. Research we publicly prioritize: see our we've prioritized or evaluated. (Also...)

2. Research we previously evaluated (see , as well as )

3. Work that other people and organizations suggest as having high potential for impact/value of information (also see)

You can also suggest research yourself and then do an independent evaluation of it.

What sort of ‘evaluations’ and what formats?

We’re looking for careful methodological/technical evaluations that focus on research credibility, impact, and usefulness. We want evaluators to dig into the weeds, particularly in areas where they have aptitude and expertise. See our.

The Unjournal’s structured evaluation forms: We encourage evaluators to do these using either:

1. Our : If you are evaluating research aimed at an academic journal or

2. Our : If you are evaluating research that is probably not aimed at an academic journal. This may include somewhat less technical work, such as reports from policy organizations and think tanks, or impact assessments and cost-benefit analyses

Other public evaluation platforms: We are also open to engaging with evaluations done on existing public evaluation platforms such as. Evaluators: If you prefer to use another platform, please let us know about your evaluation using one of the forms above. If you like, you can leave most of our fields blank, and provide a link to your evaluation on the other public platform.

Academic (~PhD) assignments and projects: We are also looking to build ties with research-intensive university programs; we can help you structure academic assignments and provide external reinforcement and feedback. Professors, instructors, and PhD students: please contact us ().

How will The Unjournal engage?

1. Posting and signal-boosting

We will encourage all these independent evaluations to be publicly hosted, and will share links to these. We will further promote the strongest independent evaluations, potentially re-hosting them on our platforms (such as )

However, when we host or link these, we will keep them clearly separated and signposted as distinct from the commissioned evaluations; independent evaluations will not be considered official, and their ratings won’t be included in our (see dashboard; see discussion).

2. Offering incentives

Bounties: We offer prizes for the 'most valuable independent evaluations'. Strong submissions may receive prizes of $500 or more.

As a reference...

All evaluation submissions will be eligible for these prizes and “grandfathered in” to any prizes announced later. We will announce and promote the prize winners (unless they opt for anonymity).

Evaluator pool: People who submit evaluations can elect to join our evaluator pool. We will consider and (time-permitting) internally rate these evaluations. People who do the strongest evaluations in our focal areas are likely to be commissioned as paid evaluators for The Unjournal.

We’re also moving towards a two-tiered base compensation for evaluations. We will offer a higher rate to people who can demonstrate previous strong review/evaluation work. These independent evaluations will count towards this ‘portfolio’.

3. Providing materials, resources and guidance/feedback

Our provides examples of strong work, including the.

We will curate guidelines and learning materials from relevant fields and from applied work and impact-evaluation. For a start, see We plan to build and curate more of this...

4. Partnering with academic institutions

We are reaching out to PhD programs and pre-PhD research-focused programs. Some curricula already involve “mock referee report” assignments. We hope professors will encourage their students to do these through our platform. In return, we’ll offer the incentives and promotion mentioned above, as well as resources, guidance, and some further feedback

5. Fostering a positive environment for anonymous and signed evaluations

We want to preserve a positive and productive environment. This is particularly important because we will be accepting anonymous content. We will take steps to ensure that the system is not abused. If the evaluations have an excessively negative tone, have content that could be perceived as personal attacks, or have clearly spurious criticism, we will ask the evaluators to revise this, or we may decide not to post or link it.

How does this benefit The Unjournal and our mission?

1. Crowdsourced feedback can add value in itself; encouraging this can enable some public evaluation and discussion of work that The Unjournal doesn’t have the bandwidth to cover

2. Improving our evaluator pool and evaluation standards in general.

   1. Students and ECRs can practice and (if possible) get feedback on independent evaluations

About The Unjournal (unfold)

commissions public evaluations of impactful research in quantitative social sciences fields. We are an alternative and a supplement to traditional academic peer-reviewed journals – separating evaluation from journals unlocks a . We ask expert evaluators to write detailed, constructive, critical reports. We also solicit a set of structured ratings focused on research credibility, methodology, careful and calibrated presentation of evidence, reasoning transparency, replicability, relevance to global priorities, and usefulness for practitioners (including funders, project directors, and policymakers who rely on this research). While we have mainly targeted impactful research from academia, our covers impactful work that uses formal quantitative methods but is not aimed at academic journals. As of January 2026, we've commissioned over 100 evaluations of 53 papers, and published these evaluation packages , linked to academic search engines and bibliometrics. See our for current statistics.

davidreinstein.org

I was in academia for about 20 years (PhD Economics, UC Berkeley; Lecturer, University of Essex; Senior Lecturer, University of Exeter). I saw how the journal system was broken.

• Academics constantly complain about it (but don't do anything to improve it).

• Most conversations are not about research, but about 'who got into what journal' and 'tricks for getting your paper into journals'

• Open science and replicability are great, and dynamic documents make research a lot more transparent and readable. But these goals and methods are very hard to apply within the traditional journal system and its 'PDF prisons'.

Now I'm working outside academia and can stick my neck out. I have the opportunity to help fix the system. I work with research organizations and large philanthropists involved with effective altruism and global priorities. They care about the results of research in areas that are relevant to global priorities. They want research to be reliable, robust, reasoning-transparent, and well-communicated. Bringing them into the equation can change the game.

Impactful research prize winners

Impactful Research Prize Winners

SFF Grant

An important part of making this a success will be to spread the word, to get positive attention for this project, to get important players on board, network externalities, and change the equilibrium. We are also looking for specific feedback and suggestions from "mainstream academics" in Economics, Psychology, and policy/program evaluation, as well as from the Open Science and EA communities.

Key points to convey

See

The Unjournal is delighted to announce the winners of our inaugural Impactful Research Prize. We are awarding our first prize to Takahiro Kubo (NIES Japan and Oxford University) and co-authors for their research titled "Banning wildlife trade can boost demand". The paper stood out for its intriguing question, the potential for policy impact, and methodological strength. We particularly appreciated the authors’ open, active, and detailed engagement with our evaluation process.

The second prize goes to Johannes Haushofer (NUS Singapore and Stockholm University) and co-authors for their work "The Comparative Impacts of Cash Transfers and a Psychotherapy Program on Psychological and Economic Wellbeing". Our evaluators rated this paper among the highest across a range of metrics. It was highly commended for its rigor, the importance of the topic, and the insightful discussion of cost-effectiveness.

We are recognizing exceptional evaluators for credible, insightful evaluations. Congratulations to Phil Trammell (Global Priorities Institute at the University of Oxford), Hannah Metzler (Complexity Science Hub Vienna), Alex Bates (independent researcher), and Robert Kubinec (NYU Abu Dhabi).

We would like to congratulate all of the winners on their contributions to open science and commitment to rigorous research. We also thank other authors who have submitted their work but have not been selected at this time - we received a lot of excellent submissions, and we are committed to supporting authors beyond this research prize.

Please see the full press release, as well as award details, below and :

\

The Unjournal was founded by David Reinstein, who maintains this GitBook and other resources.

See also: Governance of The Unjournal

Management Team

• — Founding Director

• — Co-Director (Monk Prayogshala)

• Davit Jintcharadze — Manager of Operations

Advisory Board

• — Columbia University

• — London School of Hygiene and Tropical Medicine

• — University of Melbourne

Field Specialists

We have 60+ field specialists and research affiliates organized by expertise areas including development economics, global health, behavioral science, environmental economics, AI governance, and animal welfare.

The table below shows team members taking on research-monitoring roles (see ):

Evaluator Pool

Over 200 people have signed up to assess research for The Unjournal, including approximately 90 PhD holders and 40 professors.

Interested in becoming an evaluator? .

March 25 2024: Workshop: Innovations in Research Evaluation, Replicability, and Impact

Research evaluation is changing: New approaches go beyond the traditional journal model, promoting transparency, replicability, open science, open access, and global impact. You can be a part of this.

Join us on March 25 for an interactive workshop, featuring presentations from Macie Daley (Center for Open Science), (The Unjournal), (UC Santa Barbara), and The Unjournal's Impactful Research Prize and Evaluator Prize winners.

See the for all details.

Jan 2024: Impactful Research and Evaluation Prizes winners announced

Aug. 30, 2023: "Pilot's done, what has been won (and learned)?"

Pilot = completed!

With the completed set of evaluations of and ," our pilot was complete:

• 10 research papers evaluated

• 21 evaluations

• 5 author responses

You can see all our published evaluations at .

For continuously updated statistics, see our and .

For earlier updates, see .

Sign up for our newsletter to receive occasional updates:

Alternatively, fill out this to get this newsletter and tell us some things about yourself and your interests. The data protection statement is linked .

We are not the only ones working and advocating in this space. For a small tip of the iceberg...

Unjournal evaluations aim to support the "Reproducibility/Robustness-Checking" (RRC) agenda. We are directly engaging with the Institute for Replication (I4R) and the repliCATS project (RC), and building connections to Replication Lab/TRELiSS and Metaculus.

We will support this agenda by:

1. Promoting data and code sharing: We request pre-print authors to share their code and data, and reward them for their transparency.

2. : Breaking out of "PDF prisons" to achieve increased transparency.

3. Encouraging detailed evaluations: Unjournal evaluators are asked to:

   • highlight the key/most relevant research claims, results, and tests;

   • propose possible robustness checks and tests (RRC work); and

4. Implementing computational replication and robustness checking: We aim to work with I4R and other organizations to facilitate and evaluate computational replication and robustness checking.

5. Advocating for open evaluation: We prioritize making the evaluation process transparent and accessible for all.

Research credibility

While the in psychology is well known, economics is not immune. Some very prominent and influential work has , depends on is not, or uses . Roughly 40% of experimental economics work . Prominent commenters have that the traditional journal peer-review system does a poor job of spotting major errors and identifying robust work.

Supporting the RRC agenda through Unjournal evaluations

My involvement with the shed light on a key challenge (see posts): The effectiveness of replication depends on the claims chosen for reproduction and how they are approached. I observed that it was common for the chosen claim to miss the essence of the paper, or to focus on a statistical result that, while likely to reproduce, didn't truly convey the author's message.

Simultaneously, I noticed that many papers had methodological flaws (for instance, lack of causal identification or the presence of important confounding factors in experiments). But I thought that these studies, if repeated, would likely yield similar results. These insights emerged from only a quick review of hundreds of papers and claims. This indicates that a more thorough reading and analysis could potentially identify the most impactful claims and elucidate the necessary RRC work.

Indeed, detailed, high-quality referee reports for economics journals frequently contain such suggestions. However, these valuable insights are often overlooked and rarely shared publicly. Unjournal aims to change this paradigm by focusing on three main strategies:

1. Identifying vital claims for replication:

   • We plan to have Unjournal evaluators help highlight key "claims to replicate," along with proposing replication goals and methodologies. We will flag papers that particularly need replication in specific areas.

   • Public evaluation and author responses will provide additional insight, giving future replicators more than just the original published paper to work with.

By concentrating on NBER papers, we increase the likelihood of overlap with journals targeted by the Institute for Replication, thus enhancing the utility of our evaluations in aiding replication efforts.

Our theory of change is shown above as a series of possible paths; we indicate what is arguably the most "direct" path in yellow. All of these paths begin with our setting up, funding, communicating, and incentivizing participation in a strong, open, efficient research evaluation system (in green, at the top). These processes all lead to impactful research being more in-depth, more reliable, more accessible, and more useful, and thus better informing decision-makers and leading to better decisions and outcomes (in green, at the bottom).

Highlighting some of the key paths:

(Yellow) Faster and better feedback on impactful research improves this work and better informs policymakers and philanthropists.

(Blue) Our processes and incentives will foster ties between (a) mainstream and prominent academic and policy researchers and (b) global-priorities or EA-aligned researchers. This will improve the rigor, credibility, exposure, and influence of previously "EA niche" work while helping mainstream researchers better understand and incorporate ideas, principles, and methods from the EA and rationalist research communities (such as counterfactual impact, cause-neutrality, reasoning transparency, and so on.) This process will also nudge mainstream academics towards focusing on impact and global priorities, and towards making their research and outputs more accessible and useable.

(Pink) The Unjournal’s more efficient, open, and flexible processes will become attractive to academics and stakeholders. As we become better at "predicting publication outcomes," we will become a replacement for traditional processes, improving research overall—some of which will be highly impactful research.

Detailed explanations of key paths

Rapid, informative, transparent feedback and evaluation to inform policymakers and researchers

Rigorous quantitative and empirical research in economics, business, public policy, and social science has the potential to improve our decision-making and enable a flourishing future. This can be seen in the research frameworks proposed by 80,000 Hours, Open Philanthropy, and The Global Priorities Institute (see ). This research is routinely used by effective altruists working on global priorities or existential risk mitigation. It informs both philanthropic decisions (e.g., those influenced by , whose inputs are largely based on academic research) and . Unfortunately, the academic publication process is notoriously slow; for example, in economics, it between the first presentation of a research paper and the eventual publication in a peer-reviewed journal. Recent reforms have sped up parts of the process by encouraging researchers to put working papers and preprints online.

However, working papers and preprints often receive at most only a cursory check before publication, and it is up to the reader to judge quality for themselves. Decision-makers and other researchers rely on peer review to judge the work’s credibility. This part remains slow and inefficient. Furthermore, it provides very noisy signals: A paper is typically judged by the "prestige of the journal it lands in"’ (perhaps after an intricate odyssey across journals), but it is hard to know why it ended up there. Publication success is seen to depend on personal connections, cleverness, strategic submission strategies, good presentation skills, and relevance to the discipline’s methods and theory. These factors are largely irrelevant to whether and how philanthropists and policymakers should consider and act on a paper’s claimed findings. Reviews are kept secret; the public never learns why a paper was deemed worthy of a journal, nor what its strengths and weaknesses were.

We believe that disseminating research sooner—along with measures of its credibility—is better.

We also believe that publicly evaluating its quality before (and in addition to) journal publication will add substantial additional value to the research output, providing:

1. a quality assessment (by experts in the field) that can decisionmakers and other researchers can read alongside the preprint, helping these users weigh its strengths and weaknesses and interpret its implications; and

2. faster feedback to authors focused on improving the rigor and impact of the work.

Various initiatives in the life sciences have already begun reviewing preprints. While economics took the lead in sharing working papers, public evaluation of economics, business, and social science research is rare. The Unjournal is the first initiative to publicly evaluate rapidly-disseminated work from these fields. Our specific priority: research relevant to global priorities.

Faster, better feedback; attractiveness to researchers and gatekeepers; improved research formats; and better and more useful research

The Unjournal’s open feedback should also be valuable to the researchers themselves and their research community, catalyzing progress. As the Unjournal Evaluation becomes a valuable outcome in itself, researchers can spend less time "gaming the journal system." Shared public evaluation will provide an important window to other researchers, helping them better understand the relevant cutting-edge concerns. The Unjournal will permit research to be submitted in a wider variety of useful formats (e.g., dynamic documents and notebooks rather than "frozen pdfs"), enabling more useful, replicable content and less time spent formatting papers for particular journals. We will also allow researchers to improve their work in situ and gain updated evaluations, rather than having to spin off new papers. This will make the literature more clear and less cluttered.

"Some of the main paths"

Living, "Kaizen", "permanent Beta" work

Should research projects be improved and updated 'in the same place', rather than with 'extension papers'?

Advantages of 'permanent Beta' projects

• Small changes and fixes: The current system makes it difficult to make minor updates – even obvious corrections – to published papers. This makes these papers less useful and less readable. If you find an error in your own published work, there is also little incentive to note it and ask for a correction, even if this were possible.

  • In contrast, a 'living project' could be corrected and updated in situ. If future and continued evaluations matter, they will have the incentive to do so.

• Lack of incentives for updates and extensions

But we recognize there may also be downsides to _'_all extensions and updates in a single place'...

Discussion: living projects vs the current "replication+extension approach"

1. Some discussion follows. Note that the Unjournal enables this but does not require it.

Did you just write a brilliant peer review for an economics (or social science, policy, etc.) journal? Your work should not be wasted, there should be a way to share your insights and get credit!

Consider transforming these insights into a public "independent evaluation" for . This will benefit the community and help make research better and more impactful. And we can share your work and provide you feedback. This will help you build a portfolio with The Unjournal, making it more likely we'll hire you for paid work and compensate you at the higher rate. And we offer prizes for the best work.

You can do this either anonymously or sign your name.

To say this in a fancier way:

Journal peer review is critical for assessing and improving research, but too often these valuable discussions remain hidden behind closed doors. By publishing a version of your review, you can: (1) Amplify the impact of your reviewing efforts by contextualizing the research for a broader audience, (2) Facilitate more transparent academic discussions around the strengths and limitations of the work, (3) Get public recognition for your peer review contributions, which are often unseen and unrewarded (4) Reduce overall reviewing burdens by allowing your assessment to be reused, (5) Support a culture of open scholarship by modeling constructive feedback on public research

Considerations: The reviewer "owns the review", subject to constraints and norms

According to a : Who “owns” peer reviews (emphasis added)

While the depth of commentary may vary greatly among reviews, given the minimal thresholds set by copyright law, it can be presumed that most reviews meet the requirements for protection as an “original work of authorship”. As such, in the absence of an express transfer of copyright or a written agreement between the reviewer and publisher establishing the review as a “work for hire”, it may be assumed that, by law, the reviewer holds copyright to their reviewer comments and thus is entitled to share the review however the reviewer deems fit...

The COPE council notes precisely the benefits we are aiming to unlock. They mention an 'expectation of confidentiality' that seems incompletely specified.

For example, reviewers may wish to publish their reviews in order to demonstrate their expertise in a subject matter and to contribute to their careers as a researcher. Or they may see publication of their reviews as advancing discourse on the subject and thus acting for the benefit of science as a whole. Nevertheless, a peer reviewer’s comments are significantly different from many other works of authorship in that they are expressly solicited as a work product by a journal and—whatever the peer review model—are subject to an expectation of confidentiality. However, without an express agreement between the journal and the reviewer, it is questionable whether such obligation of confidentiality should be considered to apply only until a final decision is reached on the manuscript, or to extend indefinitely.

Permission and journals' constraints

Several journals explicitly agree that reviewers are welcome to publish the content of their reviews, with some important caveats. The gathered public statements from several journals and publishers confirming that they support reviewers posting their comments externally. However, they generally ask reviewers to remove any confidential information before sharing their reviews. This includes: the name of the journal, the publication recommendation (e.g., accept, revise, or reject), and any other details the journal or authors considered confidential, such as unpublished data.

For these journals, we are happy to accept and share/link the verbatim content as part of an independent Unjournal evaluation.

But even for journals that have not signed onto this, as the COPE mentioned Your peer review is your intellectual property, it is not owned by the journal!

There may be some terms and conditions you agreed to as part of submitting a referee report. Please consult these carefully.

However, you are still entitled to share your own expert opinions on publicly-shared research. You may want to rewrite the review somewhat. You should make it clear that it refers to the publicly-shared (working paper/preprint) version of the research, not the one the journal shared with you in confidence. As above, you should probably not mention the journal name, the decision, or any other sensitive information. You don't even need to mention that you did review the paper for a journal.

Even if a journal considers the specific review confidential, this doesn't prevent the reviewer from expressing their independent assessment elsewhere.

Make a difference

As an expert reviewer, you have unique insights that can improve the quality and impact of research. Making your assessment available through The Unjournal amplifies the reach and value of your efforts. You can publish evaluations under your name or remain anonymous.

Ready to make your peer reviews work harder for science? Consider submitting an , for recognition, rewards, and to improve research. Contact us anytime at contact@unjournal.org for guidance... We look forward to unlocking your valuable insights!

\

What do we offer? How does it improve upon traditional academic review/publishing?

Reshaping academic evaluation: Beyond accept/reject: The Unjournal's process reduces the high costs and "gaming" associated with standard journal publication mechanisms.

Promoting open and robust science: We promote research replicability and robustness in line with the RRC agenda.

: We prioritize impactful work. Expert evaluators focus on reliability, robustness, and usefulness. This fosters a productive bridge between high-profile mainstream researchers and global-impact-focused organizations, researchers, and practitioners.

: We open up the evaluation process, making it more timely and transparent, and providing valuable public metrics and feedback for the benefit of authors, other researchers, and policymakers who may want to use the research..

: By separating evaluation from journal publication, free research from the static 'PDF prisons'. This enables "dynamic documents/notebooks" that boost transparency and replicability, improved research communication through web-based formats. It also facilitates "living projects": research that can continuously grow, improving in response to feedback and incorporating new data and methods in the same environment.

'Dynamic Documents' are projects or papers that are developed using prominent tools such as R-markdown or JuPyTer notebooks (the two most prominent tools).

The salient features and benefits of this approach include:

• Integrated data analysis and reporting means the data analysis (as well as math/simulations) is done and reported in the same space that the results and discussion are presented. This is made possible through the concealment of 'code blocks'.

• Transparent reporting means you can track exactly what is being reported and how it was constructed:

  • Making the process a lot less error-prone

  • Helping readers understand it better (see )

  • Helping replicators and future researchers build on it

• Other advantages of these formats (over PDFs for example) include:

  • Convenient ‘folding blocks’

  • Margin comments

Better examples, the case for dynamic documents

Reinstein's own examples

Some quick examples from my own work in progress (but other people have done it much better)

Other (randomly selected) examples

Open evaluation (and rating)

Traditional peer review is a closed process, with reviewers' and editors' comments and recommendations hidden from the public.

In contrast, all Unjournal evaluations* (along with authors' responses and evaluation manager summaries) are made public and easily accessible. We give each of these a separate DOI and work to make sure each enters the literature and bibliometric databases. We aim further to curate these, making it easy to see the evaluators' comments in the context of the research project (e.g., with sidebar/hover annotation).

Open evaluation is more useful:

• to other researchers and students (especially those early in their careers). Seeing the dialogue helps them digest the research itself and understand its relationship to the wider field. It helps them understand the strengths and weaknesses of the methods and approaches used, and how much agreement there is over these choices. It gives an inside perspective on how evaluation works.

• to people using the research, providing further perspectives on its value, strengths and weaknesses, implications, and applications.

Publicly posting evaluations and responses may also lead to higher quality and more reliability. Evaluators can choose whether or not they wish to remain anonymous; there are , but in either case, the fact that all the content is public may encourage evaluators to more fully and transparently express their reasoning and justifications. (And where they fail to do so, readers of the evaluation can take this into account.)

The fact that we are asking for evaluations and ratings of all the projects in our system—and not using "accept/reject"—should also drive more careful and comprehensive evaluation and feedback. At a traditional top-ranked journal, a reviewer may limit themselves to a few vague comments implying that the paper is "not interesting or strong enough to merit publication." This would not make sense within the context of The Unjournal.

More reliable, precise, and useful metrics

We do not "accept or reject" papers; we are evaluating research, not "publishing" it. But then, how do other researchers and students know whether the research is worth reading? How can policymakers know whether to trust it? How can it help a researcher advance their career? How can grantmakers and organizations know whether to fund more of this research?

As an alternative to the traditional measure of worth—asking, "what tier did a paper get published in?"—The Unjournal provides metrics: We ask evaluators to provide a specific set of ratings and predictions about aspects of the research, as well as aggregate measures. We make these public. We aim to synthesize and analyze these ratings in useful ways, as well as make this quantitative data accessible to meta-science researchers, meta-analysts, and tool builders.

Feel free to check out our and .

These metrics are separated into different categories designed to help researchers, readers, and users understand things like:

• How much can one believe the results stated by the authors (and why)?

• How relevant are these results for particular real-world choices and considerations?

• Is the paper written in a way that is clear and readable?

We also request overall ratings and predictions . . . of the credibility, importance, and usefulness of the work, and to help benchmark these evaluations to each other and to the current "journal tier" system.

However, even here, the Unjournal metrics are also precise in a sense that "journal publication tiers" are not. There is no agreed-upon metric of exactly how journals rank (e.g., within economics' "top-5" or "top field journals"). More importantly, there is no clear measure of the relative quality and trustworthiness of the paper within particular journals.

In addition, there are issues of lobbying, career concerns, and timing, discussed elsewhere, which make the "tiers" system less reliable. An outsider doesn't know, for example:

• Was a paper published in a top journal because of a special relationship and connections? Was an editor trying to push a particular agenda?

• Was it published in a lower-ranked journal because the author needed to get some points quickly to fill their CV for an upcoming tenure decision?

In contrast, The Unjournal requires evaluators to give specific, precise, quantified ratings and predictions (along with an explicit metric of the evaluator's uncertainty over these appraisals).

Of course, our systems will not solve all problems associated with reviews and evaluations: power dynamics, human weaknesses, and limited resources will remain. But we hope our approach moves in the right direction.

Better feedback

See also .

Faster (public) evaluation

We want to reduce the time between when research is done (and a paper or other research format is released) and when other people (academics, policymakers, journalists, etc.) have a credible measure of "how much to believe the results" and "how useful this research is."

Here's how The Unjournal can do this.

1. Early evaluation: We will evaluate potentially impactful research soon after it is released (as a working paper, preprint, etc.). We will encourage authors to submit their work for our evaluation, and we will the evaluation of work from the highest-prestige authors.

2. We will pay evaluators with further incentives for timeliness (as well as carefulness, thoroughness, communication, and insight). that these incentives for promptness and other qualities are likely to work.

3. Public evaluations and ratings: Rather than waiting years to see "what tier journal a paper lands in," the public can simply consult The Unjournal

Dynamic Documents

By “Dynamic Documents” I mean papers/projects built with Quarto, R-markdown, or JuPyTer notebooks (the most prominent tools) that do and report the data analysis (as well as math/simulations) in the same space that the results and discussion are presented (with ‘code blocks’ hidden).

I consider some of the benefits of this format, particularly for EA-aligned organizations like Open Philanthropy: Benefits of Dynamic Documents

Living Research Projects

“Continually update a project” rather than start a “new extension paper” when you see what you could have done better.

The main idea is that each version is given a specific time stamp, and that is the object that is reviewed and cited. This is more or less already the case when we cite working papers/drafts/mimeos/preprints.

See , further discussing the potential benefits.

You can fill out this form to submit your work, or email contact@unjournal.org with questions .

How does The Unjournal's evaluation process work?

We generally seek two evaluators (aka 'reviewers') with research interests in your area and with complementary expertise. You, the author, can suggest areas you want to get feedback and suggestions on.

The evaluators write detailed and helpful evaluations, and submit them either "signed" or anonymously. Using our evaluation forms, they provide quantitative ratings on several dimensions, such as methods, relevance, and communication. They predict what journal tier the research will be published in, and what tier it should be published in. Here are the .

These evaluations and ratings are typically made public (see ), but you will have the right to respond before (or after) these are posted.

To consider your research we only need a link to a publicly hosted version of your work, ideally with a DOI. We will not "publish" your paper. The fact that we are handling your paper will not limit you in any way. You can submit it to any journal before, during, or after the process.

Can I ask for evaluations to be private?

You can request a conditional embargo by emailing us at , or via the submission form. Please explain what sort of embargo you are asking for, and why. By default, we would like Unjournal evaluations to be made public promptly. However, we may make exceptions in special circumstances, particularly for very early-career researchers whose career prospects are particularly vulnerable.

If there is a vulnerable early-career researcher on the authorship team, we may allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite.

For further details on this, and some examples, see .

Do you make any requests of authors?

We may ask for some of the below, but these are mainly optional. We aim to make the process very light touch for authors.

1. A link to a non-paywalled, hosted version of your work (in any format) which can be given a DOI.

2. Responses to

We also allow you to respond to evaluations, and we give your response its own DOI.

Why should I submit my work to The Unjournal? Why should I engage with them?

By submitting your research and engaging with public evaluation, you send a powerful public signal that you are confident in your work, open to constructive criticism, and motivated to seek the truth!

And there are further benefits to authors:

1. Substantive feedback will help you improve your research. Substantive and useful feedback is often very hard to get, especially for young scholars. It's hard to get anyone to read your paper – we can help!

2. Being evaluated by The Unjournal is a sign of impact. We .

3. Ratings are markers of credibility for your work that could help your career.

Is it risky to have my work evaluated in The Unjournal?

There are risks and rewards to any activity, of course. Here we consider some risks you may weigh against the benefits mentioned above.

1. Exclusivity

2. Public negative feedback