In a nutshell

The Unjournal seeks to make rigorous research more impactful and impactful research more rigorous. We are , practitioners, and open science advocates led by .

The Unjournal encourages better research by making it easier for researchers to get feedback and credible ratings. We coordinate and fund public journal-independent evaluation of hosted papers and dynamically presented projects. We publish evaluations, ratings, manager summaries, author responses, and links to evaluated research on .

As the name suggests, we are not a journal!

We are working independently of traditional academic journals to build an open platform and a sustainable system for feedback, ratings, and assessment. We are currently focusing on quantitative work that informs global priorities in economics, policy, and other social sciences.

How to get involved?

We are looking for research papers to evaluate, as well as evaluators. If you want to suggest research, your own or someone else's, you can let us know . If you want to be an evaluator, . You can in being a member of the management team, advisory board, or reviewer pool. For more information, check our guide on . Why The Unjournal? Peer review is great, but conventional academic publication processes are wasteful, slow, and rent-extracting. They discourage innovation and prompt researchers to focus more on "gaming the system" than on the quality of their research. We will provide an immediate alternative, and at the same time, offer a bridge to a more efficient, informative, useful, and transparent research evaluation system.

Does The Unjournal charge any fees?

No. We are a nonprofit organization (hosted by ) and we do not charge any fees for submitting and evaluating your research. We compensate evaluators for their time and even award prizes for strong research work, in contrast to most traditional journals. We do so thanks to funding from the and .

At some point in the future, we might consider sliding-scale fees for people or organizations submitting their work for Unjournal evaluation, or for other services. If we do this, it would simply be a way to cover the compensation we pay evaluators and to cover our actual costs. Again, we are a nonprofit and we will stay that way.

How do we do this?

1. Research submission/identification and selection: We identify, solicit, and select relevant research work to be hosted on any open platform in any format that can gain a time-stamped DOI. Authors are encouraged to present their work in the ways they find most comprehensive and understandable. We support the use of and other formats that foster replicability and open science. (See: ).

2. Paid evaluators (AKA "reviewers"): We compensate evaluators (essentially, reviewers) for providing thorough feedback on this work. (Read more: )

Change is hard: overcoming academic inertia

Academics and funders have complained about this stuff for years . . . and we suspect our critiques of the traditional review and publication process will resonate with readers.

So why haven't academia and the research community been able to move to something new? There is a difficult collective action problem. Individual researchers and universities find it risky to move unilaterally. But we believe we have a good chance of finally changing this model and moving to a better equilibrium because we will:

• Take risks: Many members of The Unjournal management are not traditional academics; we can stick our necks out. We are also bringing on board established senior academics who are less professionally vulnerable.

• Bring in new interests, external funding, and incentives: There are a range of well-funded and powerful organizations—such as the and —with a strong inherent interest in high-impact research being reliable, robust, and . This support can fundamentally shift existing incentive structures.

Our webpage and our objectives

This GitBook serves as a platform to organize our ideas and resources and track our progress towards The Unjournal's dual objectives:

1. Making "peer evaluation and rating" of open projects into a standard high-status outcome in academia and research, specifically within economics and social sciences. This stands in contrast to the conventional binary choice of accepting or rejecting papers to be published as PDFs and other static formats.

2. Building a cohesive and efficient system for publishing, accruing credibility, and eliciting feedback for research aligned with effective altruism and global priorities. Our ultimate aim is to make rigorous research more impactful, and impactful research more rigorous.

Where do I find . . . /where do I go next?

See

Please do weigh in, all suggestions and comments will be credited. See also Unjournal: ; remember to callout contact@unjournal.org if you make any comments

Submit your work to be eligible for our “Unjournal: Impactful Research Prize” and a range of other benefits including the opportunity for credible public evaluation and feedback.

First-prize winners will be awarded $2500, and the runner-ups will receive $1000.

Note: these are the minimum amounts; we will increase these if funding permits.

Prize winners will have the opportunity (but not the obligation) to present their work at an online seminar and prize ceremony co-hosted by The Unjournal, , and

The Unjournal, with funding from the and the , organizes and funds public-journal-independent feedback and evaluation. We focus on research that is highly relevant to global priorities, especially in economics, social science, and impact evaluation, and aim to expand this widely. We encourage better research by making it easier for researchers to get feedback and credible ratings on their work.

We aim to publicly evaluate 15 papers (or projects) within our pilot year. This award will honor researchers doing robust, credible, transparent work with a global impact. We especially encourage the submission of research in "open" formats such as hosted dynamic documents (Quarto, R-markdown, Jupyter notebooks, etc.).

The research will be chosen by our management team for public evaluation by 2–3 carefully selected, paid reviewers based on an initial assessment of a paper's methodological strength, openness, clarity, relevance to , and the usefulness of further evaluation and public discussion. We sketch out .

All evaluations, including quantitative ratings, will be made public by default; however, we will consider "embargos" on this for researchers with sensitive career concerns (the linked form asks about this). Note that submitting your work to The Unjournal does not imply "publishing" it: you can submit it to any journal before, during, or after this process.

If we choose not to send your work out to reviewers, we will try to at least offer some brief private feedback (please on this).

All work evaluated by The Unjournal will be eligible for the prize. Engagement with The Unjournal, including responding to evaluator comments, will be a factor in determining the prize winners. We also have a slight preference for giving at least one of the awards to an early-career researcher, but this need not be determinative.

Our management team and advisory board will vote on the prize winners in light of the evaluations, with possible consultation of further external expertise.

Deadline: Extended until 5 December (to ensure eligibility).

Note: In a subsection below, , we outline the basic requirements for submissions to The Unjournal.

How we chose the research prize winners (2023)

The prize winners for The Unjournal's Impactful Research Prize were selected through a multi-step, collaborative process involving both the management team and the advisory board. The selection was guided by several criteria, including the quality and credibility of the research, its potential for real-world impact, and the authors' engagement with The Unjournal's evaluation process.

1. Initial Evaluation: All papers that were evaluated by The Unjournal were eligible for the prize. The discussion, evaluations, and ratings provided by external evaluators played a significant role in the initial shortlisting.

2. Management and Advisory Board Input: Members of the management committee and advisory board were encouraged to write brief statements about papers they found particularly prize-worthy.

3. Meeting and Consensus: A "prize committee" meeting was held with four volunteers from the management committee to discuss the shortlisted papers and reach a consensus. The committee considered both the papers and the content of the evaluations

This comprehensive approach aimed to ensure that the prize winners were selected in a manner that was rigorous, fair, and transparent, reflecting the values and goals of The Unjournal.

I (David Reinstein) am an economist who left UK academia after 15 years to pursue a range of projects (see my web page). One of these is The Unjournal:

The Unjournal (with funding from the Long Term Future Fund and the Survival and Flourishing Fund) organizes and funds public-journal-independent feedback and evaluation, paying reviewers for their work. We focus on research that is highly relevant to global priorities, especially in economics, social science, and impact evaluation. We encourage better research by making it easier for researchers to get feedback and credible ratings on their work.

Evaluators

We want researchers who are interested in doing evaluation work for The Unjournal. We pay an average of $400-$500 per complete and on-time evaluation, and we award monetary prizes for the strongest work. Right now we are particularly looking for economists and people with quantitative and policy-evaluation skills. We describe what we are asking evaluators to do : essentially a regular peer review with some different emphases, plus providing a set of quantitative ratings and predictions. Your evaluation content would be made public (and receive a DOI, etc.), but you can choose if you want to remain anonymous or not.

To sign up to be part of the pool of evaluators or to get involved in The Unjournal project in other ways, please or email contact@unjournal.org.

Research

We welcome suggestions for particularly impactful research that would benefit from (further) public evaluation. We choose research for public evaluation based on an initial assessment of methodological strength, openness, clarity, , and the usefulness of further evaluation and public discussion. We sketch , and discuss some potential examples (see research we have chosen and evaluated at unjournal.pubpub.org, and a larger list of research we're considering ).

If you have research—your own or others—that you would like us to assess, please fill out . You can submit your own work (or by contacting ). Authors of evaluated papers will be eligible for our (details).

Feedback

We are looking for both feedback on and involvement in The Unjournal project. Feel free to reach out at .

View our

See In a nutshell for an overview of The Unjournal.

Overview and call

I am , founder and co-director of The Unjournal. We have an open call for committee members, board members, reviewers, and suggestions for relevant work for The Unjournal to evaluate.

The is building a system for credible, public, journal-independent feedback and evaluation of research.

We maintain an open call for participants for four different roles:

1. (involving honorariums for time spent)

2. members (no time commitment)

3. (who will often also be on the Advisory Board)

You can express your interest (and enter our database) .

Evaluators

We will reach out to evaluators (a.k.a. "reviewers") on a case-by-case basis, appropriate for each paper or project being assessed. This is dependent on expertise, the researcher's interest, and a lack of conflict of interest.

Time commitment: Case-by-case basis. For each evaluation, for the amount of time to spend.

Compensation: We pay a minimum of $200 (updated Aug. 2024) for a prompt and complete evaluation, $400 for experienced evaluators. We offer additional prizes and incentives, and are committed to an average compensation of at least $450 per evaluator. .

Who we are looking for: We are putting together a list of people interested in being an evaluator and doing paid referee work for The Unjournal. We generally prioritize the pool of evaluators who signed up for our database before reaching out more widely.

Interested? Please fill out (about 3–5 min, same form for all roles or involvement).

Projects and papers

We are looking for high-quality, globally pivotal research projects to evaluate, particularly those embodying open science practices and innovative formats. We are putting out a call for relevant research. Please suggest research . (We offer bounties and prizes for useful suggestions – see note.) For details of what we are looking for, and some potential examples, and accompanying links.

You can also .

We provide a separate form for research suggestions. We may follow up with you individually.

Contact us

If you are interested in discussing any of the above in person, please email us () to arrange a conversation.

We invite you to fill in to leave your contact information, as well as outlining which parts of the project you may be interested in.

Note: This is under continual refinement; see our for more details.

Some key sections and subsections

Learn more about The Unjournal, our goals and policies

For authors, evaluators, etc.

Writeups of the main points for a few different audiences

Important benefits of journal-independent public evaluation and The Unjournal's approach, with links to deeper commentary

How we choose papers/projects to evaluate, how we assign evaluators, and so on

Other resources and reading

Groups we work with; comparing approaches

What research are we talking about? What will we cover?

Detail, progress, and internal planning

These are of more interest to people within our team; we are sharing these in the spirit of transparency.

A "best feasible plan" for going forward

Successful proposals (ACX, SFF), other applications, initiatives

Key resources and links for managers, advisory board members, staff, team members and others involved with The Unjournal project.

Note: we have moved some of this "internal interest content" over to our Coda.io knowledge base.

The potential roles discussed below combine research-linked work with operations and administrative responsibilities. Overall, this may include some combination of:

• Assisting and guiding the process of identifying strong and potentially impactful work in key areas, explaining its relevance, its strengths, and areas warranting particular evaluation and scrutiny

• Interacting with authors, recruiting, and overseeing evaluators

• Synthesizing and disseminating the results of evaluations and ratings

• Aggregating and benchmarking these results

• Helping build and improve our tools, incentives, and processes

• Curating outputs relevant to other researchers and policymakers

• Doing "meta-science" work

Possible role: Research and Evaluation Specialist (RES)

. (Nov. 2023: Note, we can not guarantee that we will be hiring for this role, because of changes in our approach.)

To indicate your potential interest in roles at The Unjournal, such as those described below, please fill out and link (or upload) your CV or webpage.

• If you already filled out this form for a role that has changed titles, don’t worry. You will still be considered for relevant and related roles in the future.

These are principally not research roles, but familiarity with research and research environments will be helpful, and there is room for research involvement depending on the candidate’s interest, background, and skills/aptitudes.

There are currently one such role:

Communications, Writing, and Public Relations Specialist (As of November 2023, still seeking freelancers)

Further note: We previously considered a “Management support and administrative professional” role. We are not planning to hire for this role currently. Those who indicated interest will be considered for other roles.

.

Communications, Writing, and Public Relations Specialist

As of November 2023, we are soliciting applications for freelancers with skills in particular areas

The Unjournal is looking to work with a proficient writer who is adept at communicating with academics and researchers (particularly in economics, social science, and policy), journalists, policymakers, and philanthropists. As we are in our early stages, this is a generalist role. We need someone to help us explain what The Unjournal does and why, make our processes easy to understand, and ensure our outputs (evaluations and research synthesis) are accessible and useful to non-specialists. We seek someone who values honesty and accuracy in communication; someone who has a talent for simplifying complex ideas and presenting them in a clear and engaging way.

The work is likely to include:

1. Promotion and general explanation

   • Spread the word about The Unjournal, our approach, our processes, and our progress in press releases and short pieces, as well as high-value emails and explanations for a range of audiences

   • Make the case for The Unjournal to potentially skeptical audiences in academia/research, policy, philanthropy, effective altruism, and beyond

Most relevant skills, aptitudes, interests, experience, and background knowledge:

• Understanding of The Unjournal project

• Strong written communications skills across a relevant range of contexts, styles, tones, and platforms (journalistic, technical, academic, informal, etc.)

• Familiarity with academia and research processes and institutions

Further desirable skills and experience:

• Academic/research background in areas related to The Unjournal’s work

• Operations, administrative, and project management experience

• Experience working in a small nonprofit institution

Proposed terms:

• Project-based contract "freelance" work

• $30–$55/hour USD (TBD, depending on experience and capabilities). Hours for each project include some onboarding and upskilling time.

• Our current budget can cover roughly 200 hours of this project work. We hope to increase and extend this (depending on our future funding and expenses).

.

In addition to the "work roles," we are looking to engage researchers, research users, meta-scientists, and people with experience in open science, open access, and management of initiatives similar to The Unjournal.

We are continually looking to enrich our general team and board, including our , These roles come with some compensation and incentives.

(Please see links and consider submitting an expression of interest).

Proposed projects

We are planning to hire 3–7 researchers for a one-off paid project.

There are two opportunities: Contracted Research (CR) and Independent Projects (IP).

Project Outline

• What specific research themes in economics, policy, and social science are most important for global priorities?

• What projects and papers are most in need of further in-depth public evaluation, attention, and scrutiny?

• Where does "Unjournal-style evaluation" have the potential to be one of the most impactful uses of time and money? By impactful, we mean in terms of some global conception of value (e.g., the well-being of living things, the survival of human values, etc.).

This is an initiative that aims to identify, summarize, and conduct an in-depth evaluation of the most impactful themes in economics, policy, and social science to answer the above questions. Through a systematic review of selected papers and potential follow-up with authors and evaluators, this project will enhance the visibility, understanding, and scrutiny of high-value research, fostering both rigorous and impactful scholarship.

Contracted Research (CR) This is the main opportunity, a unique chance to contribute to the identification and in-depth evaluation of impactful research themes in economics, policy, and social science. We’re looking for researchers and research users who can commit a (once-off) 15–20 hours. CR candidates will:

• Summarize a research area or theme, its status, and why it may be relevant to global priorities (~4 hours).

  • We are looking for fairly narrow themes. Examples might include:

    • The impact of mental health therapy on well-being in low-income countries.

We will compensate you for your time at a rate reflecting your experience and skills ($25–$65/hour). This work also has the potential to serve as a “work sample” for future roles at The Unjournal, as it is highly representative of what our and are commissioned to do.

We are likely to follow up on your evaluation suggestions. We also may incorporate your writing into our web page and public posts; you can choose whether you want to be publicly acknowledged or remain anonymous.

Independent Projects (IP)

We are also inviting applications to do similar work as an “Independent Project” (IP), a parallel opportunity designed for those eager to engage but not interested in working under a contract, or not meeting some of the specific criteria for the Contracted Research role. This involves similar work to above.

If you are accepted to do an IP, we will offer some mentoring and feedback. We will also offer prize rewards/bounties for particularly strong IP work. We will also consider working with professors and academic supervisors on these IP projects, as part of university assignments and dissertations.

You can apply to the CR and IP positions together; we will automatically consider you for each.

Get Involved!

If you are interested in involvement in either the CR or IP side of this project, please let us know .

Management committee members

Activities of those on the management committee may involve a combination of the following (although you can choose your focus):

• Contributing to the decision-making process regarding research focus, reviewer assignment, and prize distribution.

• Collaborating with other committee members on the establishment of rules and guidelines, such as determining the metrics for research evaluation and defining the mode of assessment publication.

• Helping plan The Unjournal’s future path.

Time commitment: A minimum of 15–20 hours per year.

Compensation: We have funding for a $57.50 per hour honorarium for the first 20 hours, with possible compensation for additional work. will be further compensated (at roughly $300–$450 per paper).

Who we are looking for: All applicants are welcome. We are especially interested in those involved in global priorities research (and related fields), policy research and practice, open science and meta-science, bibliometrics and scholarly publishing, and any other academic research. We want individuals with a solid interest in The Unjournal project and its goals, and the ability to meet the minimal time commitment. Applying is extremely quick, and those not chosen will be considered for other roles and work going forward.

Advisory board (AB) members

Beyond direct roles within The Unjournal, we're building a larger, more passive advisory board to be part of our network, to offer occasional feedback and guidance, and to act as an evaluation manager when needed (see our ).

There is essentially no minimum time commitment for advisory board members—only opportunities to engage. We sketch some of the expectations in the fold below.

Field specialists (FS)

FSs will focus on a particular area of research, policy, or impactful outcome. They will keep track of new or under-considered research with potential for impact and explain and assess the extent to which The Unjournal can add value by commissioning its evaluation. They will "curate" this research and may also serve as evaluation managers for this work.

Time commitment: There is no specific time obligation—only opportunities to engage. We may also consult you occasionally on your areas of expertise. Perhaps 1–4 hours a month is a reasonable starting expectation for people already involved in doing or using research, plus potential additional paid assignments.

Our document also provide some guidance on the nature of work and the time involved.

Compensation: We have put together a preliminary/trial compensation formula (); we aim to fairly compensate people for time spent on work done to support The Unjournal, and to provide incentives for suggesting and helping to prioritize research for evaluation. In addition, will be compensated at roughly $300–$450 per project.

Who we are looking for: For the FS roles, we are seeking active researchers, practitioners, and stakeholders with a strong publication record and/or involvement in the research and/or research-linked policy and prioritization processes. For the AB, also people with connections to academic, governmental, or relevant non-profit institutions, and/or involvement in open science, publication, and research evaluation processes. People who can offer relevant advice, experience, guidance, or help communicate our goals, processes, and progress.

Interested? (about 3–5 min, using the same form for all roles).

See:

Contact us

If you are interested in discussing any of the above in person, please email us () to arrange a conversation.

We invite you to (the same as that linked above) to leave your contact information and outline which parts of the project interest you.

Note: These descriptions are under continual refinement; see our for more details.

Post-pilot goals

See the vision and broad plan presented (and embedded below), updated August 2023.

Pilot targets

Updated: Partial update 10 Dec 2022.

Building research "unjournal"

See for proposed specifics.

Setup and team

✔️

✔️/⏳ Define the broad scope of our research interest and key overriding principles. Light-touch, to also be attractive to aligned academics

⏳ Build "editorial-board-like" teams with subject or area expertise

Status: Mostly completed/decided for pilot phase

Create a set of rules for "submission and management"

• Which projects enter the review system (relevance, minimal quality, stakeholders, any red lines or "musts")

  • ⏳ See for a first pass.

• How projects are to be submitted

Status: Mostly completed/decided for pilot phase; will review after initial trial

Rules for reviews/assessments

• To be done on the chosen open platform (Kotahi/Sciety) unless otherwise infeasible (10 Dec 2022 update)

• Share, advertise, promote this; have efficient meetings and presentations

  • Establish links to all open-access bibliometric initiatives (to the extent feasible)

See our .

Status: Mostly completed/decided for pilot phase; will review after the initial trial

Further steps

See our .

Status: We are still working with Google Docs and building an external survey interface. We plan to integrate this with PubPub over the coming months (August/Sept. 2023)

Note on other versions of this content.

davidreinstein.org

I was in academia for about 20 years (PhD Economics, UC Berkeley; Lecturer, University of Essex; Senior Lecturer, University of Exeter). I saw how the journal system was broken.

• Academics constantly complain about it (but don't do anything to improve it).

• Most conversations are not about research, but about 'who got into what journal' and 'tricks for getting your paper into journals'

• Open science and replicability are great, and dynamic documents make research a lot more transparent and readable. But these goals and methods are very hard to apply within the traditional journal system and its 'PDF prisons'.

Now I'm working outside academia and can stick my neck out. I have the opportunity to help fix the system. I work with research organizations and large philanthropists involved with effective altruism and global priorities. They care about the results of research in areas that are relevant to global priorities. They want research to be reliable, robust, reasoning-transparent, and well-communicated. Bringing them into the equation can change the game.

See also: Governance of The Unjournal

The Unjournal was founded by David Reinstein , who maintains this wiki/GitBook and other resources.

Management Committee

(Note on terminology)

See description under .

• , Founder and Co-director

• , Interdisciplinary Researcher at ; Co-director

• , Social Scientist and Associate Professor in the Guelph Institute of Development Studies and Department of Political Science at the University of Guelph, Canada

Advisory board

See description under .

, Infectious Disease Researcher, London School of Hygiene and Tropical Medicine

, Associate Professor of Marketing, London Business School

, Applied Researcher (Global Health & Development) at Founder's Pledge

, Professor of Economics, UC Santa Barbara

, Post-Doctoral Researcher in the Department of Quantitative Methods and Economic Theory at the University of Chieti (Italy)

, Metascience Program Lead, Federation of American Scientists

, Managing Editor at : writing and research on global health, development, and nutrition

, Professor of Statistics and Political Science at Columbia University (New York)

, Associate Professor, University of Melbourne (Australia): expert judgment, biosciences, applied probability, uncertainty quantification

, Late-Stage PhD Student in Information Systems at the University of Cologne, Germany

, PhD Student, Applied Economics, University of Minnesota

Postdoctoral researcher, Institute for Interactive Systems and Data Science at Graz University of Technology (Austria)

, Associate Researcher, INRAE, Member, Toulouse School of Economics (France)

, Data Scientist, Economist Consultant; PhD University of British Columbia (Economics)

Field Specialists

The table below shows all the members of our team (including field specialists) taking on a research-monitoring role (see for a description of this role).

Staff, contractors, and consultants

, Research Specialist: Data science, metascience, aggregation of expert judgment

, Operations generalist

, Generalist assistance

, Communications (academic research/policy)

, Communications and copy-editing

, Communications and consulting

, technical software support

Red Bermejo, Mikee Mercado, Jenny Siers – consulting (through ) on strategy, marketing, and task management tools

We are a member of . They are working with us to update PubPub and incorporate new features (editorial management, evaluation tools, etc.) that will be particularly useful to The Unjournal and other members.

Other people and initiatives we are in touch with

See also (in ACX grant proposal).

TLDR: See In a nutshell

Podcasts, presentations, and video

Journal independent evaluation and The Unjournal

EA Anywhere (Youtube) – bridging the gap between EA and academia

• See slide deck (Link: ; offers comment access)

ReproducibiliTea podcast

Slide decks

Presentation for EA Anywhere, online event, 5 Nov. 2023 1-2pm ET

(Link: ; offers comment access)

"Slaying the journals": Google doc

Earlier discussion document, aimed at EA/global priorities, academic, and open-science audiences

"

• 2021 A shorter outline posted on

EA forum posts

Impactful research prize winners

Impactful Research Prize Winners

SFF Grant

An important part of making this a success will be to spread the word, to get positive attention for this project, to get important players on board, network externalities, and change the equilibrium. We are also looking for specific feedback and suggestions from "mainstream academics" in Economics, Psychology, and policy/program evaluation, as well as from the Open Science and EA communities.

Key points to convey

See

As social media blurbs

How and where to promote and share

March 25 2024: Workshop: Innovations in Research Evaluation, Replicability, and Impact

We are not the only ones working and advocating in this space. For a small tip of the iceberg...

Progress notes since last update

Update on recent progress: 21 July 2023

Funding

The SFF grant is now 'in our account' (all is public and made transparent on our ). This makes it possible for us to

• move forward in filling staff and contractor positions (see below); and

• increase evaluator compensation and incentives/rewards (see below).

We are circulating a sharing our news and plans.

Timelines, and pipelines

Our "Pilot Phase," involving ten papers and roughly 20 evaluations, is almost complete. We just released the evaluation package for .” We are now waiting on one last evaluation, followed by author responses and then "publishing" the final two packages at . (Remember: we publish the evaluations, responses and synthesis; we link the research being evaluated.)

We will make decisions and award our (and possible seminars) and evaluator prizes soon after. The winners will be determined by a consensus of our management team and advisory board (potentially consulting external expertise). The choices will be largely driven by the ratings and predictions given by Unjournal evaluators. After we make the choices, we will make our decision process public and transparent.

"What research should we prioritize for evaluation, and why?"

We continue to develop processes and policy around which research to prioritize. For example, we are considering whether we should set targets for different fields, for related outcome "cause categories," and for research sources. This discussion continues among our team and with stakeholders. We intend to open up the discussion further, making it public and bringing in a range of voices. The objective is to develop a framework and a systematic process to make these decisions. See our expanding notes and discussion on

In the meantime, we are moving forward with our post-pilot “pipeline” of research evaluation. Our management team is considering recent prominent and influential working papers from the National Bureau of Economics Research () and beyond, and we continue to solicit submissions, suggestions, and feedback. We are also reaching out to users of this research (such as NGOs, charity evaluators, and applied research think tanks), asking them to identify research they particularly rely on and are curious about. If you want to join this conversation, we welcome your input.

(Paid) Research opportunity: to help us do this

We are also considering hiring a small number of researchers to each do a one-off (~16 hours) project in “research scoping for evaluation management.” The project is sketched at ; essentially, summarizing a research theme and its relevance, identifying potentially high-value papers in this area, choosing one paper, and curating it for potential Unjournal evaluation.

We see a lot of value in this task and expect to actually use and credit this work.

If you are interested in applying to do this paid project, please let us know .

Call for "Field Specialists"

Of course, we can't commission the evaluation of every piece of research under the sun (at least not until we get the next grant :) ). Thus, within each area, we need to find the right people to monitor and select the strongest work with the greatest potential for impact, and where Unjournal evaluations can add the most value.

This is a big task and there is a lot of ground to cover. To divide and conquer, we’re partitioning this space (looking at natural divisions between fields, outcomes/causes, and research sources) amongst our management team as well as among what we now call...

" (FSs), who will

• focus on a particular area of research, policy, or impactful outcome;

• keep track of new or under-considered research with potential for impact;

• explain and assess the extent to which The Unjournal can add value by commissioning this research to be evaluated; and

Field specialists will usually also be members of our Advisory Board, and we are encouraging expressions of interest for both together. (However, these don’t need to be linked in every case.) .

Interested in a field specialist role or other involvement in this process? Please fill out (about 3–5 minutes).

Setting priorities for evaluators

We are also considering how to set priorities for our evaluators. Should they prioritize:

• Giving feedback to authors?

• Helping policymakers assess and use the work?

• Providing a 'career-relevant benchmark' to improve research processes?

We discuss this topic , considering how each choice relates to our .

Increase in evaluator compensation, incentives/rewards

We want to attract the strongest researchers to evaluate work for The Unjournal, and we want to encourage them to do careful, in-depth, useful work. for (on-time, complete) evaluations to $400, and we are setting aside $150 per evaluation for incentives, rewards, and prizes. Details on this to come.

Please consider signing up for our evaluator pool (fill out ).

Adjacent initiatives and 'mapping this space'

As part of The Unjournal’s general approach, we keep track of (and keep in contact with) other initiatives in open science, open access, robustness and transparency, and encouraging impactful research. We want to be coordinated. We want to partner with other initiatives and tools where there is overlap, and clearly explain where (and why) we differentiate from other efforts. gives a preliminary breakdown of similar and partially-overlapping initiatives, and tries to catalog the similarities and differences to give a picture of who is doing what, and in what fields.

Also to report

New

• , Professor of Economics, UC Santa Barbara

• , Associate Researcher, INRAE, Member, Toulouse School of Economics (animal welfare agenda)

• , Associate Professor, expert judgment, biosciences, applied probability, uncertainty quantification

Tech and platforms

We're working with PubPub to improve our process and interfaces. We plan to take on a to help us work with them closely as they build their platform to be more attractive and useful for The Unjournal and other users.

Our hiring, contracting, and expansion continues

• Our next hiring focus: . We are looking for a strong writer who is comfortable communicating with academics and researchers (particularly in economics, social science, and policy), journalists, policymakers, and philanthropists. Project-based.

• We've chosen (and are in the process of contracting) a strong quantitative meta-scientist and open science advocate for the project: “Aggregation of expert opinion, forecasting, incentives, meta-science.” (Announcement coming soon.)

• We are also expanding our Management Committee and Advisory Board; see

Potentially relevant events in the outside world

Update on recent progress: 1 June 2023

Update from David Reinstein, Founder and Co-Director

A path to change

With the , we now have the opportunity to move forward and really make a difference. I think The Unjournal, along with related initiatives in other fields, should become the place policymakers, grant-makers, and researchers go to consider whether research is reliable and useful. It should be a serious option for researchers looking to get their work evaluated. But how can we start to have a real impact?

Over the next 18 months, we aim to:

1. Build awareness: (Relevant) people and organizations should know what The Unjournal is.

2. Build credibility: The Unjournal must consistently produce insightful, well-informed, and meaningful evaluations and perform effective curation and aggregation of these. The quality of our work should be substantiated and recognized.

3. Expand our scale and scope: We aim to grow significantly while maintaining the highest standards of quality and credibility. Our loose target is to evaluate around 70 papers and projects over the next 18 months while also producing other valuable outputs and metrics.

I sketch these goals , along with our theory of change, specific steps and approaches we are considering, and some "wish-list wins." Please free to add your comments and questions.

The pipeline flows on

While we wait for the new grant funding to come in, we are not sitting on our haunches. Our "pilot phase" is nearing completion. Two more sets of evaluations have been posted on our .

With three more evaluations already in progress, this will yield a total of 10 evaluated papers. Once these are completed, we will decide, announce, and award the recipients for the and the prizes for evaluators, and organize online presentations/discussions (maybe linked to an "award ceremony"?).

Contracting, hiring, expansion

No official announcements yet. However, we expect to be hiring (on a part-time contract basis) soon. This may include roles for:

• Researchers/meta-scientists: to help find and characterize research to be evaluated, identify and communicate with expert evaluators, and synthesize our "evaluation output"

• Communications specialists

• Administrative and Operations personnel

of these roles. And to indicate your potential interest and link your CV/webpage.

You can also/alternately register your interest in doing (paid) research evaluation work for The Unjournal, and/or in being part of our advisory board, .

We also plan to expand our ; please reach out if you are interested or can recommend suitable candidates.

Tech and initiatives

We are committed to enhancing our platforms as well as our evaluation and communication templates. We're also exploring strategies to nurture more beneficial evaluations and predictions, potentially in tandem with replication initiatives. A small win: our Mailchimp signup should now be working, and this update should be automatically integrated.

Welcoming new team members

We are delighted to welcome (FAS) and (INRA/TSE) to our , and (Monk Prayogshala) to our !

• Dworkin's work centers on "improving scientific research, funding, institutions, and incentive structures through experimentation."

• Treich's current research agenda largely focuses on the intersection of animal welfare and economics.

• Tagat investigates economic decision-making in the Indian context, measuring the social and economic impact of the internet and technology, and a range of other topics in applied economics and behavioral science. He is also in the .

Update on recent progress: 6 May 2023

Grant funding from the Survival and Flourishing Fund

The Unjournal was through the 'S-Process' of the Survival and Flourishing Fund. More details and plans to come. This grant will help enable The Unjournal to expand, innovate, and professionalize. We aim to build the awareness, credibility, scale, and scope of The Unjournal, and the communication, benchmarking, and useful outputs of our work. We want to have a substantial impact, building towards our mission and goals...

To make rigorous research more impactful, and impactful research more rigorous. To foster substantial, credible public evaluation and rating of impactful research, driving change in research in academia and beyond, and informing and influencing policy and philanthropic decisions.

Innovations: We are considering other initiatives and refinements (1) to our evaluation ratings, metrics, and predictions, and how these are aggregated, (2) to foster open science and robustness-replication, and (3) to provide inputs to evidence-based policy decision-making under uncertainty. Stay tuned, and please join the conversation.

Opportunities: We plan to expand our management and advisory board, increase incentives for evaluators and authors, and build our pool of evaluators and participating authors and institutions. Our previous call-to-action (see ) is still relevant if you want to sign up to be part of our evaluation (referee) pool, submit your work for evaluation, etc. (We are likely to put out a further call soon, but all responses will be integrated.)

Evaluation 'output'

We have published a total of 12 evaluations and ratings of five papers and projects, as well as three author responses. Four can be found on our PubPub page (most concise list ), and one on our Sciety page (we aim to mirror all content on both pages). All the PubPub content has a DOI, and we are working to get these indexed on Google Scholar and beyond.

The two most recently released evaluations (of Haushofer et al, 2020; and Barker et al, 2022) both surround "" [link: EA Forum post]

Both papers consider randomized controlled trials (RCTs) involving cognitive behavioral therapy (CBT) for low-income households in two African countries (Kenya and Ghana). These papers come to very different conclusions as to the efficacy of this intervention.

See the evaluation summaries and ratings, with linked evaluations and

Update on recent progress: 22 April 2023

New 'output'

We are now up to twelve total evaluations of five papers. Most of these are on our (we are currently aiming to have all of the work hosted both at PubPub and on Sciety, and gaining DOIs and entering the bibliometric ecosystem). The latest two are on an interesting theme, :

Two more Unjournal Evaluation sets are out. Both papers consider randomized controlled trials (RCTs) involving cognitive behavioral therapy (CBT) for low-income households in two African countries (Kenya and Ghana). These papers come to very different conclusions as to the efficacy of this intervention.

These are part of Unjournal's .

More evaluations coming out soon on themes including global health and development, the environment, governance, and social media.

Animal welfare

To round out our initial pilot: We're particularly looking to evaluate papers and projects relevant to animal welfare and animal agriculture. Please reach out if you have suggestions.

New features of this GitBook: GPT-powered 'chat' Q&A

You can now 'chat' with this page, ask questions, and get answers with links to other parts of the page. To try it out, go to "Search" and choose "Lens."

Update on recent progress: 17 Mar 2023

See our latest post on the EA Forum

1. Our new platform (), enabling DOIs and CrossRef (bibliometrics)

2. "self-correcting science"

3. More evaluations soon

Update on recent progress: 19 Feb 2023

Content and 'publishing'

1. Our is up...

2. With our ("Long Term Cost-Effectiveness of Resilient Foods"... Denkenberger et al. Evaluations from Scott Janzwood, Anca Hanea, and Alex Bates, and an author response.

3. Two more evaluations 'will be posted soon' (waiting for final author responses.

Tip of the Spear ... right now we are:

• Working on getting six further papers (projects) evaluated, most of which are part of our NBER

• Developing and discussing tools for aggregating and presenting the evaluators' quantitative judgments

• Building our platforms, and considering ways to better format and integrate evaluations

Funding, plans, collaborations

We are seeking grant funding for our continued operation and expansion (see below). We're appealing to funders interested in Open Science and in impactful research.

We're considering collaborations with other compatible initiatives, including...

• replication/reproducibility/robustness-checking initiatives,

• prediction and replication markets,

• and projects involving the elicitation and 'aggregation of expert and stakeholder beliefs' (about both replication and outcomes themselves).

Management and administration, deadlines

• We are now under the 'fiscal sponsorship' (this does not entail funding, only a legal and administrative home). We are postponing the deadline for judging the and the prizes for evaluators. Submission of papers and the processing of these has been somewhat slower than expected.

Other news and media

• EA Forum: "recent post and AMA (answering questions about the Unjournal's progress, plans, and relation to effective-altruism-relevant research

• March 9-10: David Reinstein will present at the , session on "Translating Open Science Best Practices to Non-academic Settings". See . David will discuss The Unjournal for part of this session.

Calls to action

See: . These are basically still all relevant.

1. Evaluators: We have a strong pool of evaluators.

Recall, we pay at least $250 per evaluation, we typically pay more in net ($350), and we are looking to increase this compensation further. Please fill out (about 3-5 min) if you are interested

1. Research to evaluate/prizes: We continue to be interested in submitted and suggested work. One area we would like to engage with more: quantitative social science and economics work relevant to animal welfare.

Hope these updates are helpful. Let me know if you have suggestions.

The Unjournal is delighted to announce the winners of our inaugural Impactful Research Prize. We are awarding our first prize to Takahiro Kubo (NIES Japan and Oxford University) and co-authors for their research titled "Banning wildlife trade can boost demand". The paper stood out for its intriguing question, the potential for policy impact, and methodological strength. We particularly appreciated the authors’ open, active, and detailed engagement with our evaluation process.

The second prize goes to Johannes Haushofer (NUS Singapore and Stockholm University) and co-authors for their work "The Comparative Impacts of Cash Transfers and a Psychotherapy Program on Psychological and Economic Wellbeing". Our evaluators rated this paper among the highest across a range of metrics. It was highly commended for its rigor, the importance of the topic, and the insightful discussion of cost-effectiveness.

We are recognizing exceptional evaluators for credible, insightful evaluations. Congratulations to Phil Trammell (Global Priorities Institute at the University of Oxford), Hannah Metzler (Complexity Science Hub Vienna), Alex Bates (independent researcher), and Robert Kubinec (NYU Abu Dhabi).

We would like to congratulate all of the winners on their contributions to open science and commitment to rigorous research. We also thank other authors who have submitted their work but have not been selected at this time - we received a lot of excellent submissions, and we are committed to supporting authors beyond this research prize.

Please see the full press release, as well as award details, below and :