1 of 7

Action and progress

The steps we've taken and our plans; needs updating

This page and its sub-pages await updating

Gantt Chart of next steps

18 Jun 2023: This needs updating

Initial evaluations; feedback on the process
Revise process; further set of evaluations
Disseminate and communicate (research, evaluations, processes); get further public feedback
Further funding; prepare for scaling-up

Management: updates and CTA in gdoc shared in emails

Pilot steps

See sections below.

Pilot: Building a founding committee

7 Feb 2023: We have an organized founding/management committee, as well as an advisory board (see Our team). We are focusing on pushing research through the evaluation pipeline, communicating this output, and making it useful. We have a working division of labor, e.g., among "managing editors," for specific papers. We are likely to expand our team after our pilot, conditional on further funding.

Progress: the team (continual update)

Key elements of plan

Put together founding committee, meetings, public posts, and feedback (done)

Build a "founding committee" of 5–8 experienced and enthusiastic EA-aligned or adjacent researchers at EA orgs, research academics, and practitioners (e.g., draw from speakers at recent EA Global meetings).
1. Create private Airtable with lists of names and organizations

How was this founding committee recruited?

The creation of an action plan can be seen in the Gdoc discussion

Three key relevant areas from which to draw candidates

DR: I think I need to draw people from a few relevant areas: 1. Academia, in relevant subject fields for The Unjournal: economics, quantitative social science, maybe more

2. Effective altruism, to assess the value and scope of the journal and the research

3. Open Science and academic reform, and applied metascience—people with practical ideas and knowledge

+ People with strong knowledge of the journal and bibliometric processes and systems

First: direct outreach to a list of pivotal, prominent people

Assemble a list of the most relevant and respected people, using more or less objective criteria and justification.
1. Ask to join founding committee.
2. Ask to join list of supporters.

28 May 2022: The above has mostly been done, at least in terms of people attending the first meeting. We probably need a more systematic approach to getting the list of supporters.

Second: public call for interest

Further posts on social media, academic websites and message boards, etc.

Pilot: Identifying key research

(Note for the Management Team)

Test-case research for proof of concept

Identify a small set of papers or projects as representative first-cases; use to help test the system we are building in a concrete manner.

In doing the above, we are also collecting a longer list of key papers, projects, authors, topics, issues, etc.

Steps taken

Post on EA Forum (and other places) and present (see view at bottom of this section) promoting our call for papers further, with bounty.

2. Search for most-cited papers (within our domain) among EA-aligned researchers and organizations.

3. Dig into existing lists, reviews, and syllabi, such as:

GPI (includes many posed questions)
Open Philanthropy (needs updating? few academic links)
Syllabi:

Consider: "Mistakes" pages?

(mainly operational mistakes)

Not very relevant because focused on operational issues

5. Appeal directly to authors and research groups

6. Cross-promote with

Pivot: direct focus on NBER working papers

Pete Slattery: "Do a complete test run using a single paper and team…" Thus, we aim to identify a small set of papers (or projects), maybe 2–3, that seem like good test and example cases, and offer a bounty for projects we choose as test cases.
Note that much of the work identified here has already been peer-reviewed and "published." While we envision that The Unjournal may assess papers that are already published in traditional journals, these are probably not the best case for the PoC. Thus, we de-prioritize these for now.

Pilot: Setting up platforms

Set up the basic platforms for posting and administering reviews and evaluations and offering curated links and categorizations of papers and projects.

Progress reports

Update 7 Sep 2022, partial update 22 Dec 2022

We are setting up processes and forms in
- is pretty useable (but imperfect, e.g., we need to ask people to (click 'submit a URL instead' on page one)
Evaluations form: using a Gdoc for now, trying out Airtable, Qualtrics and other solutions, aiming to integrate it into Kotahi
See for how projects will enter, be evaluated, and 'output'
We will outline specific for developers\
Sciety group set up with 'Hypothes.is feed'; working on processing first evaluations\

Submission, evaluation and management platform

7 Feb 2023

Set up
Configured it for submission and management
Mainly configured for evaluation but it needs bespoke configuration to be efficient and easy for evaluators, particular for the quantitative ratings and predictions. Thus we are using Google Docs (or cryptpads) for the pilot. Will configure Kotahi with further funds.

Sciety group (curated evaluations and research)

Evaluations are curated , which integrates these with the publicly-hosted research.

7 Feb 2023: We are working on

The best ways to get evaluations from "submissions on Kotahi" into Sciety,
... with the curated link to the publicly-hosted papers (or projects) on a range of platforms, including NBER
Ways to get DOIs for each evaluation and author response
Ways to integrate evaluation details as 'collaborative annotations' (with hypothes.is) into the hosted papers

(We currently use a hypothes.is workaround to have this feed into so these show up as ‘evaluated pre-prints’ in their public database, gaining a DOI.

Setting up evaluation guidelines for pilot papers

7 Feb 2023 We have considered and put together:

See: Guidelines for evaluators

... including descriptive and quantitative (rating and prediction elements). With feedback from evaluators and others, we are continuing to build and improve these guidelines.

'Evaluators': Identifying and engaging

Status: 7 Feb 2023

Volunteer pool of 80+ reviewers (see Airtable), responding to How to get involved and other outreach
For our initial 3 focal pilot papers we have a total of 8 completed evaluations (2 of these are completed subject to some final checks. Status tracked in AirtableReviewer_process in participants_reviewers
For the remaining 7 pilot papers, we have roughly 6 agreed evaluators so far (we aim for 2-3 per paper)

Pilot: Identifying key research

(Note for the Management Team)

Test-case research for proof of concept

Identify a small set of papers or projects as representative first-cases; use to help test the system we are building in a concrete manner.

In doing the above, we are also collecting a longer list of key papers, projects, authors, topics, issues, etc.

Steps taken

Post on EA Forum (and other places) and present (see view at bottom of this section) promoting our call for papers further, with bounty.

2. Search for most-cited papers (within our domain) among EA-aligned researchers and organizations.

3. Dig into existing lists, reviews, and syllabi, such as:

GPI (includes many posed questions)
Open Philanthropy (needs updating? few academic links)
Syllabi:

Consider: "Mistakes" pages?

(mainly operational mistakes)

Not very relevant because focused on operational issues

5. Appeal directly to authors and research groups

6. Cross-promote with

Pivot: direct focus on NBER working papers

Pete Slattery: "Do a complete test run using a single paper and team…" Thus, we aim to identify a small set of papers (or projects), maybe 2–3, that seem like good test and example cases, and offer a bounty for projects we choose as test cases.
Note that much of the work identified here has already been peer-reviewed and "published." While we envision that The Unjournal may assess papers that are already published in traditional journals, these are probably not the best case for the PoC. Thus, we de-prioritize these for now.

Pilot: Setting up platforms

Set up the basic platforms for posting and administering reviews and evaluations and offering curated links and categorizations of papers and projects.

Progress reports

Update 7 Sep 2022, partial update 22 Dec 2022

We are setting up processes and forms in
- is pretty useable (but imperfect, e.g., we need to ask people to (click 'submit a URL instead' on page one)
Evaluations form: using a Gdoc for now, trying out Airtable, Qualtrics and other solutions, aiming to integrate it into Kotahi
See for how projects will enter, be evaluated, and 'output'
We will outline specific for developers\
Sciety group set up with 'Hypothes.is feed'; working on processing first evaluations\

Submission, evaluation and management platform

7 Feb 2023

Set up
Configured it for submission and management
Mainly configured for evaluation but it needs bespoke configuration to be efficient and easy for evaluators, particular for the quantitative ratings and predictions. Thus we are using Google Docs (or cryptpads) for the pilot. Will configure Kotahi with further funds.

Sciety group (curated evaluations and research)

Evaluations are curated , which integrates these with the publicly-hosted research.

7 Feb 2023: We are working on

The best ways to get evaluations from "submissions on Kotahi" into Sciety,
... with the curated link to the publicly-hosted papers (or projects) on a range of platforms, including NBER
Ways to get DOIs for each evaluation and author response
Ways to integrate evaluation details as 'collaborative annotations' (with hypothes.is) into the hosted papers

(We currently use a hypothes.is workaround to have this feed into so these show up as ‘evaluated pre-prints’ in their public database, gaining a DOI.

Pilot: Building a founding committee

Progress: the team (continual update)

Key elements of plan

Put together founding committee, meetings, public posts, and feedback (done)

Build a "founding committee" of 5–8 experienced and enthusiastic EA-aligned or adjacent researchers at EA orgs, research academics, and practitioners (e.g., draw from speakers at recent EA Global meetings).
1. Create private Airtable with lists of names and organizations

How was this founding committee recruited?

The creation of an action plan can be seen in the Gdoc discussion

Three key relevant areas from which to draw candidates

DR: I think I need to draw people from a few relevant areas: 1. Academia, in relevant subject fields for The Unjournal: economics, quantitative social science, maybe more

2. Effective altruism, to assess the value and scope of the journal and the research

3. Open Science and academic reform, and applied metascience—people with practical ideas and knowledge

+ People with strong knowledge of the journal and bibliometric processes and systems

First: direct outreach to a list of pivotal, prominent people

Assemble a list of the most relevant and respected people, using more or less objective criteria and justification.
1. Ask to join founding committee.
2. Ask to join list of supporters.

28 May 2022: The above has mostly been done, at least in terms of people attending the first meeting. We probably need a more systematic approach to getting the list of supporters.

Second: public call for interest

Further posts on social media, academic websites and message boards, etc.

Action and progress

Gantt Chart of next steps

Pilot steps

Pilot: Building a founding committee

Progress: the team (continual update)

Key elements of plan

How was this founding committee recruited?

First: direct outreach to a list of pivotal, prominent people

Second: public call for interest

See also public Gdoc

Pilot: Identifying key research

Test-case research for proof of concept

Steps taken

Pivot: direct focus on NBER working papers

Pilot: Setting up platforms

Submission, evaluation and management platform

Sciety group (curated evaluations and research)

Setting up evaluation guidelines for pilot papers

'Evaluators': Identifying and engaging

Pilot: Identifying key research

Test-case research for proof of concept

Steps taken

Pivot: direct focus on NBER working papers

Setting up evaluation guidelines for pilot papers

'Evaluators': Identifying and engaging

Pilot: Setting up platforms

Submission, evaluation and management platform

Sciety group (curated evaluations and research)

Action and progress

Gantt Chart of next steps

Pilot: Building a founding committee

Progress: the team (continual update)

Key elements of plan

How was this founding committee recruited?

First: direct outreach to a list of pivotal, prominent people

Second: public call for interest

See also public Gdoc

Pilot steps