To indicate your potential interest in roles at The Unjournal, such as those described below, please fill out this quick survey form and link (or upload) your CV or webpage.

• If you already filled out this form for a role that has changed titles, don’t worry. You will still be considered for relevant and related roles in the future.

• If you add your name to this form, we may contact you to offer you the opportunity to do paid project work and paid work tasks.

Furthermore, if you are interested in conducting paid research evaluation for The Unjournal, or in joining our advisory board, please complete the form linked .

Feel free to contact contact@unjournal.org with any questions.

Quick links to role descriptions below

Additional information

The Unjournal, a not-for-profit collective under the umbrella and fiscal sponsorship of the , is an equal-opportunity employer and contractor. We are committed to creating an inclusive environment for all employees, volunteers, and contractors. We do not discriminate on the basis of race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetic information, disability, age, or veteran status.

See our data protection statement .

I (David Reinstein) am an economist who left UK academia after 15 years to pursue a range of projects (see my web page). One of these is The Unjournal:

The Unjournal (with funding from the Long Term Future Fund and the Survival and Flourishing Fund) organizes and funds public-journal-independent feedback and evaluation, paying reviewers for their work. We focus on research that is highly relevant to global priorities, especially in economics, social science, and impact evaluation. We encourage better research by making it easier for researchers to get feedback and credible ratings on their work.

Evaluators

We want researchers who are interested in doing evaluation work for The Unjournal. We pay an average of $400-$500 per complete and on-time evaluation, and we award monetary prizes for the strongest work. Right now we are particularly looking for economists and people with quantitative and policy-evaluation skills. We describe what we are asking evaluators to do : essentially a regular peer review with some different emphases, plus providing a set of quantitative ratings and predictions. Your evaluation content would be made public (and receive a DOI, etc.), but you can choose if you want to remain anonymous or not.

To sign up to be part of the pool of evaluators or to get involved in The Unjournal project in other ways, please or email contact@unjournal.org.

Research

We welcome suggestions for particularly impactful research that would benefit from (further) public evaluation. We choose research for public evaluation based on an initial assessment of methodological strength, openness, clarity, , and the usefulness of further evaluation and public discussion. We sketch , and discuss some potential examples (see research we have chosen and evaluated at unjournal.pubpub.org, and a larger list of research we're considering ).

If you have research—your own or others—that you would like us to assess, please fill out . You can submit your own work (or by contacting ). Authors of evaluated papers will be eligible for our (details).

Feedback

We are looking for both feedback on and involvement in The Unjournal project. Feel free to reach out at .

View our

These are principally not research roles, but familiarity with research and research environments will be helpful, and there is room for research involvement depending on the candidate’s interest, background, and skills/aptitudes.

There are currently one such role:

(As of November 2023, still seeking freelancers)

Further note: We previously considered a “Management support and administrative professional” role. We are not planning to hire for this role currently. Those who indicated interest will be considered for other roles.

See for an overview of The Unjournal.

Overview and call

I am , founder and co-director of The Unjournal. We have an open call for committee members, board members, reviewers, and suggestions for relevant work for The Unjournal to evaluate.

The is building a system for credible, public, journal-independent feedback and evaluation of research.

We maintain an open call for participants for four different roles:

1. (involving honorariums for time spent)

2. members (no time commitment)

3. (who will often also be on the Advisory Board)

You can express your interest (and enter our database) .

Evaluators

We will reach out to evaluators (a.k.a. "reviewers") on a case-by-case basis, appropriate for each paper or project being assessed. This is dependent on expertise, the researcher's interest, and a lack of conflict of interest.

Time commitment: Case-by-case basis. For each evaluation, for the amount of time to spend.

Compensation: We pay a minimum of $200 (updated Aug. 2024) for a prompt and complete evaluation, $400 for experienced evaluators. We offer additional prizes and incentives, and are committed to an average compensation of at least $450 per evaluator. .

Who we are looking for: We are putting together a list of people interested in being an evaluator and doing paid referee work for The Unjournal. We generally prioritize the pool of evaluators who signed up for our database before reaching out more widely.

Interested? Please fill out (about 3–5 min, same form for all roles or involvement).

Projects and papers

We are looking for high-quality, globally pivotal research projects to evaluate, particularly those embodying open science practices and innovative formats. We are putting out a call for relevant research. Please suggest research . (We offer bounties and prizes for useful suggestions – see note.) For details of what we are looking for, and some potential examples, and accompanying links.

You can also .

We provide a separate form for research suggestions. We may follow up with you individually.

Contact us

If you are interested in discussing any of the above in person, please email us () to arrange a conversation.

We invite you to fill in to leave your contact information, as well as outlining which parts of the project you may be interested in.

Note: This is under continual refinement; see our for more details.

Submit your work to be eligible for our “Unjournal: Impactful Research Prize” and a range of other benefits including the opportunity for credible public evaluation and feedback.

First-prize winners will be awarded $2500, and the runner-ups will receive $1000.

Note: these are the minimum amounts; we will increase these if funding permits.

Prize winners will have the opportunity (but not the obligation) to present their work at an online seminar and prize ceremony co-hosted by The Unjournal, , and

The Unjournal, with funding from the and the , organizes and funds public-journal-independent feedback and evaluation. We focus on research that is highly relevant to global priorities, especially in economics, social science, and impact evaluation, and aim to expand this widely. We encourage better research by making it easier for researchers to get feedback and credible ratings on their work.

We aim to publicly evaluate 15 papers (or projects) within our pilot year. This award will honor researchers doing robust, credible, transparent work with a global impact. We especially encourage the submission of research in "open" formats such as hosted dynamic documents (Quarto, R-markdown, Jupyter notebooks, etc.).

The research will be chosen by our management team for public evaluation by 2–3 carefully selected, paid reviewers based on an initial assessment of a paper's methodological strength, openness, clarity, relevance to , and the usefulness of further evaluation and public discussion. We sketch out .

All evaluations, including quantitative ratings, will be made public by default; however, we will consider "embargos" on this for researchers with sensitive career concerns (the linked form asks about this). Note that submitting your work to The Unjournal does not imply "publishing" it: you can submit it to any journal before, during, or after this process.

If we choose not to send your work out to reviewers, we will try to at least offer some brief private feedback (please on this).

All work evaluated by The Unjournal will be eligible for the prize. Engagement with The Unjournal, including responding to evaluator comments, will be a factor in determining the prize winners. We also have a slight preference for giving at least one of the awards to an early-career researcher, but this need not be determinative.

Our management team and advisory board will vote on the prize winners in light of the evaluations, with possible consultation of further external expertise.

Deadline: Extended until 5 December (to ensure eligibility).

Note: In a subsection below, , we outline the basic requirements for submissions to The Unjournal.

How we chose the research prize winners (2023)

The prize winners for The Unjournal's Impactful Research Prize were selected through a multi-step, collaborative process involving both the management team and the advisory board. The selection was guided by several criteria, including the quality and credibility of the research, its potential for real-world impact, and the authors' engagement with The Unjournal's evaluation process.

1. Initial Evaluation: All papers that were evaluated by The Unjournal were eligible for the prize. The discussion, evaluations, and ratings provided by external evaluators played a significant role in the initial shortlisting.

2. Management and Advisory Board Input: Members of the management committee and advisory board were encouraged to write brief statements about papers they found particularly prize-worthy.

3. Meeting and Consensus: A "prize committee" meeting was held with four volunteers from the management committee to discuss the shortlisted papers and reach a consensus. The committee considered both the papers and the content of the evaluations

This comprehensive approach aimed to ensure that the prize winners were selected in a manner that was rigorous, fair, and transparent, reflecting the values and goals of The Unjournal.

Proposed projects

We are planning to hire 3–7 researchers for a one-off paid project.

There are two opportunities: Contracted Research (CR) and Independent Projects (IP).

Project Outline

• What specific research themes in economics, policy, and social science are most important for global priorities?

• What projects and papers are most in need of further in-depth public evaluation, attention, and scrutiny?

• Where does "Unjournal-style evaluation" have the potential to be one of the most impactful uses of time and money? By impactful, we mean in terms of some global conception of value (e.g., the well-being of living things, the survival of human values, etc.).

This is an initiative that aims to identify, summarize, and conduct an in-depth evaluation of the most impactful themes in economics, policy, and social science to answer the above questions. Through a systematic review of selected papers and potential follow-up with authors and evaluators, this project will enhance the visibility, understanding, and scrutiny of high-value research, fostering both rigorous and impactful scholarship.

Contracted Research (CR) This is the main opportunity, a unique chance to contribute to the identification and in-depth evaluation of impactful research themes in economics, policy, and social science. We’re looking for researchers and research users who can commit a (once-off) 15–20 hours. CR candidates will:

• Summarize a research area or theme, its status, and why it may be relevant to global priorities (~4 hours).

  • We are looking for fairly narrow themes. Examples might include:

    • The impact of mental health therapy on well-being in low-income countries.

We will compensate you for your time at a rate reflecting your experience and skills ($25–$65/hour). This work also has the potential to serve as a “work sample” for future roles at The Unjournal, as it is highly representative of what our and are commissioned to do.

We are likely to follow up on your evaluation suggestions. We also may incorporate your writing into our web page and public posts; you can choose whether you want to be publicly acknowledged or remain anonymous.

Independent Projects (IP)

We are also inviting applications to do similar work as an “Independent Project” (IP), a parallel opportunity designed for those eager to engage but not interested in working under a contract, or not meeting some of the specific criteria for the Contracted Research role. This involves similar work to above.

If you are accepted to do an IP, we will offer some mentoring and feedback. We will also offer prize rewards/bounties for particularly strong IP work. We will also consider working with professors and academic supervisors on these IP projects, as part of university assignments and dissertations.

You can apply to the CR and IP positions together; we will automatically consider you for each.

Get Involved!

If you are interested in involvement in either the CR or IP side of this project, please let us know .

Note on other versions of this content.

Initiative: ‘independent evaluations’

is seeking academics, researchers, and students to submit structured evaluations of the most impactful research emerging in the social sciences. Strong evaluations will be posted or linked on our , offering readers a perspective on the implications, strengths, and limitations of the research. These evaluations can be submitted using for academic-targeted research or for more applied work; evaluators can publish their name or maintain anonymity; we also welcome collaborative evaluation work. We will facilitate, promote, and encourage these evaluations in several ways, described below.

Who should do these evaluations?

We are particularly looking for people with research training, experience, and expertise in quantitative social science and statistics including cost-benefit modeling and impact evaluation. This could include professors, other academic faculty, postdocs, researchers outside of academia, quantitative consultants and modelers, PhD students, and students aiming towards PhD-level work (pre-docs, research MSc students etc.) But anyone is welcome to give this a try — when in doubt, piease go for it.

We are also happy to support collaborations and group evaluations. There is a good track record for this — see: “”, ASAPBio’s, and for examples in this vein. We may also host live events and/or facilitate asynchronous collaboration on evaluations

Instructors/PhD, MRes, Predoc programs: We are also keen to work with students and professors to integrate ‘independent evaluation assignments’ (aka ‘learn to do peer reviews’) into research training.

Why should you do an evaluation?

Your work will support The Unjournal’s core mission — improving impactful research through journal-independent public evaluation. In addition, you’ll help research users (policymakers, funders, NGOs, fellow researchers) by providing high quality detailed evaluations that rate and discuss the strengths, limitations, and implications of research.

Doing an independent evaluation can also help you. We aim to provide feedback to help you become a better researcher and reviewer. We’ll also give prizes for the strongest evaluations. Lastly, writing evaluations will help you build a portfolio with The Unjournal, making it more likely we will commission you for paid evaluation work in the future.

Which research?

We focus on rigorous, globally-impactful research in quantitative social science and policy-relevant research. (See for details.) We’re especially eager to receive independent evaluations of:

1. Research we publicly prioritize: see our we've prioritized or evaluated. (Also...)

2. Research we previously evaluated (see , as well as )

You can also suggest research yourself and then do an independent evaluation of it.

What sort of ‘evaluations’ and what formats?

We’re looking for careful methodological/technical evaluations that focus on research credibility, impact, and usefulness. We want evaluators to dig into the weeds, particularly in areas where they have aptitude and expertise. See our.

The Unjournal’s structured evaluation forms: We encourage evaluators to do these using either:

1. Our : If you are evaluating research aimed at an academic journal or

2. Our : If you are evaluating research that is probably not aimed at an academic journal. This may include somewhat less technical work, such as reports from policy organizations and think tanks, or impact assessments and cost-benefit analyses

Other public evaluation platforms: We are also open to engaging with evaluations done on existing public evaluation platforms such as. Evaluators: If you prefer to use another platform, please let us know about your evaluation using one of the forms above. If you like, you can leave most of our fields blank, and provide a link to your evaluation on the other public platform.

Academic (~PhD) assignments and projects: We are also looking to build ties with research-intensive university programs; we can help you structure academic assignments and provide external reinforcement and feedback. Professors, instructors, and PhD students: please contact us ().

How will The Unjournal engage?

1. Posting and signal-boosting

We will encourage all these independent evaluations to be publicly hosted, and will share links to these. We will further promote the strongest independent evaluations, potentially re-hosting them on our platforms (such as )

However, when we host or link these, we will keep them clearly separated and signposted as distinct from the commissioned evaluations; independent evaluations will not be considered official, and their ratings won’t be included in our (see dashboard; see discussion).

2. Offering incentives

Bounties: We will offer prizes for the ‘most valuable independent evaluations’.

As a start, after the first eight quality submissions (or by Jan. 1 2025, whichever comes later), we will award a prize of $500 to the most valuable evaluation.

Further details tbd. As a reference...

All evaluation submissions will be eligible for these prizes and “grandfathered in” to any prizes announced later. We will announce and promote the prize winners (unless they opt for anonymity).

Evaluator pool: People who submit evaluations can elect to join our evaluator pool. We will consider and (time-permitting) internally rate these evaluations. People who do the strongest evaluations in our focal areas are likely to be commissioned as paid evaluators for The Unjournal.

We’re also moving towards a two-tiered base compensation for evaluations. We will offer a higher rate to people who can demonstrate previous strong review/evaluation work. These independent evaluations will count towards this ‘portfolio’.

3. Providing materials, resources and guidance/feedback

Our provides examples of strong work, including the.

We will curate guidelines and learning materials from relevant fields and from applied work and impact-evaluation. For a start, see We plan to build and curate more of this...

4. Partnering with academic institutions

We are reaching out to PhD programs and pre-PhD research-focused programs. Some curricula already involve “mock referee report” assignments. We hope professors will encourage their students to do these through our platform. In return, we’ll offer the incentives and promotion mentioned above, as well as resources, guidance, and some further feedback.

How does this benefit The Unjournal and our mission?

1. Crowdsourced feedback can add value in itself; encouraging this can enable some public evaluation and discussion of work that The Unjournal doesn’t have the bandwidth to cover

2. Improving our evaluator pool and evaluation standards in general.

   1. Students and ECRs can practice and (if possible) get feedback on independent evaluations

About The Unjournal (unfold)

commissions public evaluations of impactful research in quantitative social sciences fields. We are an alternative and a supplement to traditional academic peer-reviewed journals – separating evaluation from journals unlocks a . We ask expert evaluators to write detailed, constructive, critical reports. We also solicit a set of structured ratings focused on research credibility, methodology, careful and calibrated presentation of evidence, reasoning transparency, replicability, relevance to global priorities, and usefulness for practitioners (including funders, project directors, and policymakers who rely on this research).[2] While we have mainly targeted impactful research from academia, our covers impactful work that uses formal quantitative methods but is not aimed at academic journals. So far, we’ve commissioned about 50 evaluations of 24 papers, and published these evaluation packages , linked to academic search engines and bibliometrics.

The potential roles discussed below combine research-linked work with operations and administrative responsibilities. Overall, this may include some combination of:

• Assisting and guiding the process of identifying strong and potentially impactful work in key areas, explaining its relevance, its strengths, and areas warranting particular evaluation and scrutiny

• Interacting with authors, recruiting, and overseeing evaluators

• Synthesizing and disseminating the results of evaluations and ratings

• Aggregating and benchmarking these results

• Helping build and improve our tools, incentives, and processes

• Curating outputs relevant to other researchers and policymakers

• Doing "meta-science" work

Possible role: Research and Evaluation Specialist (RES)

. (Nov. 2023: Note, we can not guarantee that we will be hiring for this role, because of changes in our approach.)