For research authors

How does The Unjournal's evaluation process work?

We generally seek (aka reviewers or referees) with research interests in your area and with complementary expertise. You, the author, can suggest areas that you want to on.

The evaluators write detailed and helpful evaluations, and submit them either "signed" or anonymously. They provide quantitative ratings on several dimensions, such as methods, relevance, and communication. They predict what journal tier the research will be published in, and what tier it should be published in. Here are the Guidelines for evaluators.

These evaluations and ratings are typically made public (see unjournal.pubpub.org), but you will have the right to respond before or after these are posted.

To consider your research we only need a link to a publicly hosted version of your work, ideally with a DOI. We will not "publish" your paper. The fact that we are handling your paper will not limit you in any way. You can submit it to any journal before, during, or after the process.

Can I ask for evaluations to be private?

You can request a conditional embargo by emailing us at [e-mail], or via the submission form. Please explain what sort of embargo you are asking for, and why. By default, we would like Unjournal evaluations to be made public promptly. However, we may make exceptions in special circumstances, particularly for very early-career researchers.

If there is an early-career researcher on the authorship team, we may allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite.

For further details on this, and examples, see "Conditional embargoes and exceptions".

Do you make any requests of authors?

We may ask for some of the below, but these are mainly optional. We aim to make the process very light touch for authors.

1. A link to a non-paywalled, hosted version of your work (in ) which .

2. Responses to .

3. We may ask for a link to data and code, if possible. Note that our project is not principally about replication, and we are not insisting on this.

We also allow you to respond to evaluations, and we give your response its own DOI.

Why should I submit my work to The Unjournal? Why should I engage with them?

The biggest personal gains for :

1. Substantive feedback will help you improve your research. Substantive and useful feedback is often very hard to get, especially for young scholars. It's hard to get anyone to read your paper – we can help!

2. Ratings are markers of credibility for your work that could help your career. Remember, we select our research based on potential global relevance.

3. The chance to publicly respond to criticism and correct misunderstandings.

4. Increasing the visibility of your work, which may lead to additional citations. We publicize our evaluations and the original papers on our social media feed. We may forward selected work to Asterisk magazine.

5. A connection to the EA/Global Priorities communities and the Open Science movement. This may lead to grant opportunities, open up new ambitious projects, and attract strong PhD students to your research groups.

6. A reputation as an early adopter and innovator in open science.

7. Prizes: You may win an Impactful Research Prize (pilot) which will be financial as well as reputational.

8. Undervalued or updated work: Your paper may have been "under-published". Perhaps there are a limited set of prestigious journals in your field. You now see ways you could improve the research. The Unjournal can help; we will consider post-publication evaluation. (See fold.)

9. Innovative formats: Journals typically require you to submit a LaTeX or MS Word file, and to use their fussy formats and styles. You may want to use tools like Quarto that integrate your code and data, allow you to present dynamic content, and enhance reproducibility. The Unjournal , and we can evaluate research in virtually any format.

Unjournal evaluations may particularly help you if...

Your work may be "under-published" because of:

Time pressure: Perhaps you, or a co-author were in a hurry and submitted it to a "safe" but low-ranked journal.

Interdisciplinary approaches: It may have been considered by a journal in one field, but it needs feedback and credibility from other fields (e.g., theory vs. empirics, etc.).

Improvements: You may have substantially improved and extended the work since its publication. The current journal system can only handle this if you claim it is a 'brand new paper'. We aim to fix this (see Benefits of Dynamic Documents)

Limited journal opportunities: You may have "used up" the good journals in your field, or otherwise fallen through the cracks.

Your focus on reliability and impact: Perhaps your work paper is very impactful and the empirics are strong, but the field doesn't see it as innovative or clever. You need another opportunity.

Is it risky to have my work evaluated in The Unjournal?

There are risks and rewards to any activity, of course. Here we consider some risks you may weigh against the benefits mentioned above.

1. Exclusivity

2. Public negative feedback

Exclusivity

, and perhaps they might enforce these more strongly if they fear competition from The Unjournal.

However, The Unjournal is not exclusive. Having your paper reviewed and evaluated in The Unjournal will not limit your options; you can still submit your work to traditional journals.

Public negative feedback

Our evaluations are public. While there has been some movement towards open review, this is still not standard. Typically when you submit your paper, reviews are private. With The Unjournal, you might get public negative evaluations.

We think this is an acceptable risk. Most academics expect that opinions will differ about a piece of work, and everyone has received negative reviews. Thus, getting public feedback — in The Unjournal or elsewhere — should not particularly harm you or your research project.

Nonetheless, we are planning some exceptions for early-career researchers (see Can I ask for evaluations to be private?).

Unjournal evaluations should be seen as signals of research quality. Like all such signals, they are noisy. But submitting to The Unjournal shows you are confident in your work, and not afraid of public feedback.

How will these 'signals' be seen? (discussion)

Authors might worry that "a bad signal will hurt a lot, and a good signal will only help a little."

But if getting any public feedback was so damaging, why would academics be eager to present their work at seminars and conferences?

The main point: Unbiased signals cannot systematically lead to beliefs updating in one direction.

For fancy and formal ways of saying this and related concepts, see Martingales, Conservation of Expected Evidence, Rational Expectations, and the Law of Iterated Expectations.

In other words, people will take your transparency into account.

If a reviewer evaluates a paper without much information on how others rated it, they might suspect that there have been some substantial and important criticisms and that the paper has flaws. This is particularly the case if it has been around a while, they are submitting to a second-tier journal, etc.

On the other hand, if I see that an author has submitted their work to a transparent assessment service like The Unjournal I would update positively, all else being equal. I would consider the public critiques, of course, but I will take into account that this paper has been held up to more scrutiny than other work I might have seen.

I received the message "Your paper is being evaluated by The Unjournal". What does this mean?

Within our "Direct evaluation" track, The Unjournal directly chooses papers (from prominent archives, well-established researchers, etc.) to evaluate. We don't request authors' permission here.

As you can see in our evaluation workflow, on this track, we engage with authors at (at least) two points:

1. Informing the authors that the evaluation will take place, requesting they engage, and giving them the opportunity to request a conditional embargo or specific types of feedback.

   • Of particular interest: are we looking at the most recent version of the paper/project, or is there a further revised version we should be considering instead?

2. After the evaluations have been completed, the authors are given two weeks to respond, and have their response posted along with our 'package'. (Authors can also respond after we have posted the evaluations, and we will put their response in the same 'package', with a DOI etc.)

I received the message "The Unjournal can confirm we have received your submitted manuscript." What does this mean?

Once we receive unsolicited work from an author or authors, we keep it in our database and have our team decide on prioritization. If your paper is prioritized for evaluation, The Unjournal will notify you.

At present, we do not have a system to fully share the status of author submissions with authors. We hope to have one in place in the near future.

You can still submit your work to any traditional journal.

What if we have revised the paper since the version evaluated by The Unjournal?

The Unjournal aims to evaluate the most recent version of a paper. We reach out to authors to ensure we have the latest version at the start of the evaluation process.

If substantial updates are made to a paper during the evaluation process, authors are encouraged to share the updated version. We then inform our evaluators and ask if they wish to revise their comments.

If the evaluators can't or don't respond, we will note this and still link the newest version.

Authors are encouraged to respond to evaluations by highlighting major revisions made to their paper, especially those relevant to the evaluators' critiques. If authors are not ready to respond to evaluations, we can post a placeholder response indicating that responses and/or a revised version of the paper are forthcoming.

Re-evaluation: If authors and evaluators are willing to engage, The Unjournal is open to re-evaluating a revised version of a paper after publishing the evaluations of the initial version.

Why should I provide an "author's response" to evaluations?

We share evaluations with the authors and give them a chance to respond before we make the evaluations public (and again afterward, at any point). We add these to our evaluation packages on PubPub. Evaluation manager's (public) reports and our further communications incorporate the paper, the evaluations, and the authors' responses.

Authors' responses could bring several benefits...

1. Personally: a chance to correct misconceptions and explain their approach and planned steps. If you spot any clear errors or omissions, we give evaluators a chance to adjust their reports in response. The authors response can also help others (including the evaluators, as well as journal referees and grant funders) to have a more accurate and positive view of the research

2. For research users, to get an informed balanced perspective on how to judge the work

3. For other researchers, to better understand the methodological issues and approaches. This can serve to start a public dialogue and discussion to help build the field and research agenda. Ultimately, we aim to facilitate a back-and-forth between authors, evaluators, and others.

What should an 'author response' look like?

Examples: We've received several detailed and informative author responses, such as:

• Gerster and Epperson, “Willful Ignorance and Moral Behavior”

• Denkenberger et al on "Long term cost-effectiveness of resilient foods..."

• Trammell and Aschenbrenner, “Existential Risk and Growth”

Evaluations may raise substantive doubts and questions, and make some specific suggestions, and ask about (e.g.) data, context, or assumptions. There's no need to respond to every evaluator point; only you have something substantive: clarifying doubts, explaining the justification for your particular choices, and giving your thoughts on the suggestions (which will you incorporate, or not, and why?).

A well-written author response would (ideally) have a clear narrative and group responses into themes.

Try to have a positive tone (no personal attacks etc.) but avoid too much formality, politeness, or flattery. Revise-and-resubmit responses at standard journals sometimes begin each paragraph with "thank you for this excellent suggestion". Feel free to skip this; we want to focus on the substance.