We generally seek two evaluators (aka reviewers or referees) with research interests in your area and with complementary expertise. You, the author, can suggest areas that you want to receive feedback and suggestions on.
The evaluators write detailed and helpful evaluations, and submit them either "signed" or anonymously. They provide quantitative ratings on several dimensions, such as methods, relevance,and communication. They predict what journal tier the research will be published in, and what tier it should be published in. Here are the .
These evaluations and ratings are typically made public (see ), but you will have the right to respond before or after these are posted.
To consider your research we only need a link to a publicly hosted version of your work, ideally with a DOI. We will not "publish" your paper. The fact that we are handling your paper will not limit you in any way. You can submit it to any journal before, during, or after the process.
Can I ask for evaluations to be private?
You can request a conditional embargo by emailing us at , or via the submission form. Please explain what sort of embargo you are asking for, and why. By default, we would like Unjournal evaluations to be made public promptly. However, we may make exceptions in special circumstances, particularly for very early-career researchers.
If there is an early-career researcher on the authorship team, we may allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite. For example, we might grant an embargo that lasts until after a PhD/postdoc’s upcoming job market or until after publication in a mainstream journal, with a hard maximum of 14 months. (Of course, embargoes can be ended early at the request of the authors.)
In exceptional circumstances we may consider granting a "conditional indefinite embargo."
Do you make any requests of authors?
We may ask for some of the below, but these are mainly optional. We aim to make the process very light touch for authors.
A link to a non-paywalled, hosted version of your work (in any format) which can be given a DOI.
We also allow you to respond to evaluations, and we give your response its own DOI.
Why should I submit my work to The Unjournal? Why should I engage with them?
The biggest personal gains for authors are:
Substantive feedback will help you improve your research. Substantive and useful feedback is often very hard to get, especially for young scholars. It's hard to get anyone to read your paper – we can help!
Ratings are markers of credibility for your work that could help your career. Remember, we .
The chance to publicly respond to criticism and correct misunderstandings.
Unjournal evaluations may particularly help you if...
Your work may be "under-published" because of:
Time pressure: Perhaps you, or a co-author were in a hurry and submitted it to a "safe" but low-ranked journal.
Interdisciplinary approaches: It may have been considered by a journal in one field, but it needs feedback and credibility from other fields (e.g., theory vs. empirics, etc.).
Improvements: You may have substantially improved and extended the work since its publication. The current journal system can only handle this if you claim it is a 'brand new paper'. We aim to fix this (see
Is it risky to have my work evaluated in The Unjournal?
There are risks and rewards to any activity, of course. Here we consider some risks you may weigh against the benefits mentioned above.
Exclusivity
Public negative feedback
Exclusivity
Some traditional journals might have restrictions on the public sharing of your work, and perhaps they might enforce these more strongly if they fear competition from The Unjournal.
However, The Unjournal is not exclusive. Having your paper reviewed and evaluated in The Unjournal will not limit your options; you can still submit your work to traditional journals.
Public negative feedback
Our evaluations are public. While there has been some movement towards open review, this is still not standard. Typically when you submit your paper, reviews are private. With The Unjournal, you might get public negative evaluations.
We think this is an acceptable risk. Most academics expect that opinions will differ about a piece of work, and everyone has received negative reviews. Thus, getting public feedback — in The Unjournal or elsewhere — should not particularly harm you or your research project.
Nonetheless, we are planning some exceptions for early-career researchers (see ).
Unjournal evaluations should be seen as . Like all such signals, they are noisy. But submitting to The Unjournal shows you are confident in your work, and not afraid of public feedback.
How will these 'signals' be seen? (discussion)
Authors might worry that "a bad signal will hurt a lot, and a good signal will only help a little."
But if getting any public feedback was so damaging, why would academics be eager to present their work at seminars and conferences?
The main point:Unbiased signals cannot systematically lead to beliefs updating in one direction.
For fancy and formal ways of saying this and related concepts, see , , Rational Expectations, and the Law of Iterated Expectations.
I received the message "Your paper is being evaluated by The Unjournal". What does this mean?
Within our , The Unjournal directly chooses papers (from prominent archives, well-established researchers, etc.) to evaluate. We don't request authors' permission here.
As you can see in our , on this track, we engage with authors at (at least) two points:
Informing the authors that the evaluation will take place, requesting they engage, and giving them the opportunity to request a or specific types of feedback.
Of particular interest: are we looking at the most recent version of the paper/project, or is there a further revised version we should be considering instead?
After the evaluations have been completed, the authors are given two weeks to respond, and have their response posted along with our 'package'. (Authors can also respond after we have posted the evaluations, and we will put their response in the same 'package', with a DOI etc.)
I received the message "The Unjournal can confirm we have received your submitted manuscript." What does this mean?
Once we receive unsolicited work from an author or authors, we keep it in our database and have our team decide on prioritization. If your paper is prioritized for evaluation, The Unjournal will notify you.
At present, we do not have a system to fully share the status of author submissions with authors. We hope to have one in place in the near future.
You can still submit your work to any traditional journal.
What if we have revised the paper since the version evaluated by The Unjournal?
The Unjournal aims to evaluate the most recent version of a paper. We reach out to authors to ensure we have the latest version at the start of the evaluation process.
If substantial updates are made to a paper during the evaluation process, authors are encouraged to share the updated version. We then inform our evaluators and ask if they wish to revise their comments.
If the evaluators can't or don't respond, we will note this and still link the newest version.
Authors are encouraged to respond to evaluations by highlighting major revisions made to their paper, especially those relevant to the evaluators' critiques. If authors are not ready to respond to evaluations, we can post a placeholder response indicating that responses and/or a revised version of the paper are forthcoming.
Re-evaluation: If authors and evaluators are willing to engage, The Unjournal is open to re-evaluating a revised version of a paper after publishing the evaluations of the initial version.
Why should I provide an "author's response" to evaluations?
We share evaluations with the authors and give them a chance to respond before we make the evaluations public (and again afterward, at any point). We add these to our . Evaluation manager's (public) reports and our further communications incorporate the paper, the evaluations, and the authors' responses.
Authors' responses could bring several benefits...
Personally: a chance to correct misconceptions and explain their approach and planned steps. If you spot any clear errors or omissions, we give evaluators a chance to adjust their reports in response. The authors response can also help others (including the evaluators, as well as journal referees and grant funders) to have a more accurate and positive view of the research
For research users, to get an informed balanced perspective on how to judge the work
For other researchers, to better understand the methodological issues and approaches. This can serve to start a public dialogue and discussion to help build the field and research agenda. Ultimately, we aim to facilitate a back-and-forth between authors, evaluators, and others.
What should an 'author response' look like?
Examples: We've received several detailed and informative author responses, such as:
Evaluations may raise substantive doubts and questions, and make some specific suggestions, and ask followup questions about (e.g.) data, context, or assumptions. There's no need to respond to every evaluator point; only you have something substantive: clarifying doubts, explaining the justification for your particular choices, and giving your thoughts on the suggestions (which will you incorporate, or not, and why?).
A well-written author response would (ideally) have a clear narrative and group responses into themes.
Try to have a positive tone (no personal attacks etc.) but avoid too much formality, politeness, or flattery. Revise-and-resubmit responses at standard journals sometimes begin each paragraph with "thank you for this excellent suggestion". Feel free to skip this; we want to focus on the substance.
We may ask for a link to data and code, if possible. Note that our project is not principally about replication, and we are not insisting on this.
Increasing the visibility of your work, which may lead to additional citations. We publicize our evaluations and the original papers on our social media feed. We may forward selected work to Asterisk magazine for coverage.
A connection to the EA/Global Priorities communities and the Open Science movement. This may lead to grant opportunities, open up new ambitious projects, and attract strong PhD students to your research groups.
A reputation as an early adopter and innovator in open science.
Undervalued or updated work: Your paper may have been "under-published". Perhaps there are a limited set of prestigious journals in your field. You now see ways you could improve the research. The Unjournal can help; we will consider post-publication evaluation. (See fold.)
Innovative formats: Journals typically require you to submit a LaTeX or MS Word file, and to use their fussy formats and styles. You may want to use tools like Quarto that integrate your code and data, allow you to present dynamic content, and enhance reproducibility. The Unjournal encourages this, and we can evaluate research in virtually any format.
)
Limited journal opportunities: You may have "used up" the good journals in your field, or otherwise fallen through the cracks.
Your focus on reliability and impact: Perhaps your work paper is very impactful and the empirics are strong, but the field doesn't see it as innovative or clever. You need another opportunity.
In other words, people will take your transparency into account.
If a reviewer evaluates a paper without much information on how others rated it, they might suspect that there have been some substantial and important criticisms and that the paper has flaws. This is particularly the case if it has been around a while, they are submitting to a second-tier journal, etc.
On the other hand, if I see that an author has submitted their work to a transparent assessment service like The Unjournal I would update positively, all else being equal. I would consider the public critiques, of course, but I will take into account that this paper has been held up to more scrutiny than other work I might have seen.
We refer to "evaluation" because The Unjournal does not publish work; it only links, rates, and evaluates it.
For more information about what we are asking evaluators (referees) to do, see:
Choosing and working with evaluators
How do we choose evaluators?
We follow standard procedures, considering complementary expertise, interest, and cross-citations, as well as confirming no conflicts of interest. (See our internal guidelines for .)
We aim to consult those who have first.
We favor evaluators with a track record of careful, in-depth, and insightful evaluation — while giving ECRs a chance to build such a record.
Why do we pay evaluators?
For several reasons... (for more discussion, see )
It's equitable, especially for those not getting "service credit" for their refereeing work from their employer.
Paying evaluators can reduce adverse selection and conflicts of interest —arguably inherent to the traditional process where reviewers work for free.
To claim your evaluator payment...
Evaluator concerns
Can I submit an evaluation anonymously? How will you protect my anonymity?
Yes, we allow evaluators to choose whether they wish to remain anonymous or "sign" their evaluations. See .
I'm concerned about making my evaluation public; what if I make a mistake or write something I later regret?
To limit this concern:
You can choose to make your evaluation anonymous. You can make this decision from the outset (this is preferable) or later, after you've completed your review.
Your evaluation will be shared with the authors before it is posted, and they will be given two weeks to respond before we post. If they cite what they believe are any major misstatements in your evaluation, we will give you the chance to correct these.
It is well-known that referee reports and evaluations are subject to mistakes. We expect most people who read your evaluation will take this into account.
Can I redact my evaluation after it's published through The Unjournal?
We will put your evaluation on and give it a DOI. It cannot be redacted in the sense that this initial version will remain on the internet in some format. But you can add an addendum to the document later, which we will post and link, and the DOI can be adjusted to point to the revised version.
Are the research authors involved in The Unjournal's review process and do they give consent?
See the FAQ as well as the .
We have two main ways that papers and research projects enter the Unjournal process:
Authors submit their work; if we believe the work is relevant, we assign evaluators, and so on.
We select research that seems potentially influential, impactful, and relevant for evaluation. In some cases, we request the authors' permission before sending out the papers for evaluation. In other cases (such as where senior authors release papers in the prestigious
For either track, authors are invited to be involved in several ways:
Authors are informed of the process and given an opportunity to identify particular concerns, request an embargo, etc.
Evaluators can be put in touch with authors (anonymously) for clarification questions.
Authors are given a two-week window to respond to the evaluations (this response is published as well) before the evaluations are made public. They can also respond after the evaluations are released.
Can I share this evaluation? What else can I do with it?
If you are writing a signed evaluation, you can share it or link it on your own pages. Please wait to do this until after we have given the author a chance to respond and posted the package.
Otherwise, if you are remaining anonymous, please do not disclose your connection to this report.
Going forward:
We may later invite you to write and evaluate more about this piece of research . . .
. . . and to help us judge prizes (e.g., the ).
Evaluation value
What value do these evaluations provide, and for whom?
The evaluations provide at least three types of value, helping advance several paths in our :
For readers and users: Unjournal evaluations assess the reliability and usefulness of the paper along several dimensions—and make this public, so other researchers and policymakers can learn from them.
For careers and improving research: Evaluations provide metrics of quality. In the medium term, these should provide increased and accelerated career value, improving the research process. We aim to build metrics that are credibly comparable to the current "tier" of journal a paper is published in. But we aim to do this better in several ways:
See "what
Evaluation quality and misunderstandings
What should I prioritize in my evaluation process?
See our .
'This paper is great, I would accept it without changes, what should I write/do?"
We still want your evaluation and ratings. Some things to consider as an evaluator in this situation.
A paper/project is not only a good to be judged on a single scale. How useful is it, and to who or what? We'd like you discuss its value in relation to previous work, it’s implications, what it suggests for research and practice, etc.
Even if the paper is great...
Would you accept it in the “top journal in economics”? If not, why not?
Would you hire someone based on this paper?
Would you fund a major intervention (as a government policymaker, major philanthropist, etc.) based on this paper alone? If not, why not
We need to use explicit incentives while The Unjournal grows.
We can use payment as an incentive for high-quality work, and to access a wider range of expertise, including people not interested in submitting their own work to The Unjournal.
You can add an addendum or revision to your evaluation later on (see below).
and CEPR series) we contact the authors and request their engagement before proceeding, but we don't ask for permission.
We may ask if you want to be involved in replication exercises (e.g., through the ).
As a general principle, we hope and intend always to see that you are fairly compensated for your time and effort.
More quickly, more reliably, more transparently, and without the unproductive overhead of dealing with journals (see '')
Allowing flexible, , thus improving the research process, benefiting research careers, and hopefully improving the research itself in impactful areas.
Feedback and suggestions for authors: We expect that evaluators will provide feedback that is relevant to the authors, to help them make the paper better.
What are the most important and informative results of the paper?
Can you quantify your confidence in these 'crucial' results, and their replicability and generalizability to other settings? Can you state your probabilistic bounds (confidence or credible intervals) on the quantitative results (e.g., 80% bounds on QALYs/DALYs/or WELLBYs per $1000)
Would any other robustness checks or further work have the potential to increase your confidence (narrow your belief bounds) in this result? Which?
Do the authors make it easy to reproduce the statistical (or other) results of the paper from shared data? Could they do more in this respect?
Communication: Did you understand all of the paper? Was it easy to read? Are there any parts that could have been better explained?
Is it communicated in a way that would it be useful to policymakers? To other researchers in this field, or in the general discipline?
You can now ask questions of this GitBook using a chatbot! Click the search bar and choose 'ask gitbook'.
General FAQs
What is The Unjournal?
We organize and fund public, journal-independent feedback, rating, and evaluation of academic work. We focus on work that is highly relevant to global priorities, especially in economics, social science, and impact evaluation. We encourage better research by making it easier for researchers to get credible feedback. See for more details.
Does The Unjournal charge fees?
No. The Unjournal does not charge any fees. In fact, unlike most traditional journals, we compensate evaluators for their time, and award prizes for strong work.
We are a nonprofit organization. We do not charge fees for access to our evaluations, and work to make them as open as possible.
In future, we may consider sliding-scale fees for people submitting their work for Unjournal evaluation. If so, this would simply be to cover our costs and compensate evaluators. We are a nonprofit and will stay that way.
Is The Unjournal a journal?
No. We do not publish research. We just commission public evaluation and rating of relevant research that is already publicly hosted. Having your work evaluated in The Unjournal should not limit you from submitting it to any publication.
How is The Unjournal funded? Is it sustainable?
We have from philanthropists and organizations who are interested in our priority research areas. We hope that our work will provide enough value to justify further direct funding. We may also seek funding from governments and universities supporting the open-access agenda.