Links to EA Forum/"EA journal"

This initiative and EA/gp Unjournal will interact with the EA forum and build on initiatives coming there.

Some of these links come from a conversation with Aaron Gertler

EA Forum suggestions and formats

Here's where to suggest new Forum features.
Here's an example of a PR FAQ post that led us to develop a new feature.

Note: Reinstein and Hamish Huggard have worked on tools to help transform R-markdown and bookdown files. Some work can be found on this Repo (but may need some explanation).

Peer review on the EA Forum?

Jaime Sevilla has thoughts on creating a peer-review system for the Forum. (See embedded doc below, link here.)

Peter Slattery

To create a quick and easy prototype to test, you fork the EA Forum and use that fork as a platform for the Unjournal project (maybe called something like "The Journal of Social Impact Improvement and Assessment").
People (ideally many from EA) would use the Forum-like interface to submit papers to this Unjournal.
These papers would look like EA Forum posts, but with an included OSF link to a PDF version. Any content (e.g., slides or video) could be embedded in the submission.
All submissions would be reviewed by a single admin (you?) for basic quality standards.
Most drafts would be accepted to The Unjournal.
Any accepted drafts would be publicly "peer reviewed." They would achieve peer-reviewed status when >x (3?) people from a predetermined or elected board of editors or experts had publicly or anonymously reviewed the paper by commenting publicly on the post. Reviews might also involve ratings the draft on relevant criteria (INT?). Public comment/review/rating would also be possible.
Draft revisions would be optional but could be requested. These would simply be new posts with version X/v X appended to the title.
All good comments or posts to the journal would receive upvotes, etc., so authors, editors and commentators would gain recognition, status and "points" from participation. This is sufficient for generating participation in most forums and notably lacking in most academic settings.
Good papers submitted to the journal would be distinguished by being more widely read, engaged with, and praised than others. If viable, they would also win prizes. As an example, there might be a call for papers on solving issue x with a reward pool of grant/unconditional funding of up to $x for winning submissions. The top x papers submitted to The Unjournal in response to that call would get grant funding for further research.
A change in rewards/incentives (from "I had a paper accepted/cited" to "I won a prize") seems to have various benefits.
It still works for traditional academic metrics—grant money is arguably even more prized than citations and publication in many settings
It works for non-academics who don't care about citations or prestigious journal publications.
As a metric, "funds received" would probably better track researchers' actual impact than their citations and acceptance in a top journal. People won't pay for more research that they don't value, but they will cite or accept that to a journal for other reasons.
Academics could of course still cite the DOIs and get citations tracked this way.
Reviewers could be paid per-review by research commissioners.
Here is a quick example of how it could work for the first run: Open Philanthropy calls for research on something they want to know about (e.g., interventions to reduce wild animal suffering). They commit to provide up $100,000 in research funding for good submissions and $10,000 for review support. Ten relevant experts apply and are elected to the expert editorial boards to review submissions. They will receive 300 USD per review and are expected to review at least x papers. People submit papers; these are reviewed; OP awards follow-up prizes to the winning papers. The cycle repeats with different funders, and so on.
I suppose I like the above because it seems pretty easy and actionable to do over as a test run for something to refine and scale. I estimate that I could probably do it myself if I had 6–12 months to focus on it. However, I imagine that I am missing a few key considerations as I am usually over-optimistic! Feel free to point those out and offer feedback.

PreviousEA forum peer reviewing (related)NextOther non-journal evaluation

Last updated 1 year ago

Peer review *on* the EA Forum?

Jaime Sevilla has thoughts on creating a peer-review system for the Forum. (See embedded doc below, link here.)

Peter Slattery

To create a quick and easy prototype to test, you fork the EA Forum and use that fork as a platform for the Unjournal project (maybe called something like "The Journal of Social Impact Improvement and Assessment").
People (ideally many from EA) would use the Forum-like interface to submit papers to this Unjournal.
These papers would look like EA Forum posts, but with an included OSF link to a PDF version. Any content (e.g., slides or video) could be embedded in the submission.
All submissions would be reviewed by a single admin (you?) for basic quality standards.
Most drafts would be accepted to The Unjournal.
Any accepted drafts would be publicly "peer reviewed." They would achieve peer-reviewed status when >x (3?) people from a predetermined or elected board of editors or experts had publicly or anonymously reviewed the paper by commenting publicly on the post. Reviews might also involve ratings the draft on relevant criteria (INT?). Public comment/review/rating would also be possible.
Draft revisions would be optional but could be requested. These would simply be new posts with version X/v X appended to the title.
All good comments or posts to the journal would receive upvotes, etc., so authors, editors and commentators would gain recognition, status and "points" from participation. This is sufficient for generating participation in most forums and notably lacking in most academic settings.
Good papers submitted to the journal would be distinguished by being more widely read, engaged with, and praised than others. If viable, they would also win prizes. As an example, there might be a call for papers on solving issue x with a reward pool of grant/unconditional funding of up to $x for winning submissions. The top x papers submitted to The Unjournal in response to that call would get grant funding for further research.
A change in rewards/incentives (from "I had a paper accepted/cited" to "I won a prize") seems to have various benefits.
It still works for traditional academic metrics—grant money is arguably even more prized than citations and publication in many settings
It works for non-academics who don't care about citations or prestigious journal publications.
As a metric, "funds received" would probably better track researchers' actual impact than their citations and acceptance in a top journal. People won't pay for more research that they don't value, but they will cite or accept that to a journal for other reasons.
Academics could of course still cite the DOIs and get citations tracked this way.
Reviewers could be paid per-review by research commissioners.
Here is a quick example of how it could work for the first run: Open Philanthropy calls for research on something they want to know about (e.g., interventions to reduce wild animal suffering). They commit to provide up $100,000 in research funding for good submissions and $10,000 for review support. Ten relevant experts apply and are elected to the expert editorial boards to review submissions. They will receive 300 USD per review and are expected to review at least x papers. People submit papers; these are reviewed; OP awards follow-up prizes to the winning papers. The cycle repeats with different funders, and so on.

I suppose I like the above because it seems pretty easy and actionable to do over as a test run for something to refine and scale. I estimate that I could probably do it myself if I had 6–12 months to focus on it. However, I imagine that I am missing a few key considerations as I am usually over-optimistic! Feel free to point those out and offer feedback.