You can now ask questions of this GitBook using a chatbot! and choose 'ask gitbook'.
We organize and fund public, journal-independent feedback, rating, and evaluation of academic work. We focus on work that is highly relevant to global priorities, especially in economics, social science, and impact evaluation. We encourage better research by making it easier for researchers to get credible feedback. See here for more details.
No. The Unjournal does not charge any fees. In fact, unlike most traditional journals, we compensate evaluators for their time, and award prizes for strong work.
We are a nonprofit organization. We do not charge fees for access to our evaluations, and work to make them as open as possible. In future, we may consider sliding-scale fees for people submitting their work for Unjournal evaluation. If so, this would simply be to cover our costs and compensate evaluators. We are a nonprofit and will stay that way.
No. We do not research. We just commission public evaluation and rating of relevant research that is already . Having your work evaluated in The Unjournal from submitting it to any publication.
We have grants from philanthropists and organizations who are interested in our priority research areas. We hope that our work will provide enough value to justify further direct funding. We may also seek funding from governments and universities supporting the open-access agenda.
Sure! Please contact us at theunjournal@gmail.com.
because The Unjournal does not publish work; it only links, rates, and evaluates it.
For more information about what we are asking evaluators (referees) to do, see:
We follow standard procedures, considering complementary expertise, interest, and cross-citations, as well as confirming no conflicts of interest. (See our internal guidelines for .)
We aim to consult those who have first.
We favor evaluators with a track record of careful, in-depth, and insightful evaluation — while giving ECRs a chance to build such a record.
It's equitable, especially for those not getting "service credit" for their refereeing work from their employer.
We need to use explicit incentives while The Unjournal grows.
We can use payment as an incentive for high-quality work, and to access a wider range of expertise, including people not interested in submitting their own work to The Unjournal.
To limit this concern:
You can choose to make your evaluation anonymous. You can make this decision from the outset (this is preferable) or later, after you've completed your review.
Your evaluation will be shared with the authors before it is posted, and they will be given two weeks to respond before we post. If they cite what they believe are any major misstatements in your evaluation, we will give you the chance to correct these.
It is well-known that referee reports and evaluations are subject to mistakes. We expect most people who read your evaluation will take this into account.
You can add an addendum or revision to your evaluation later on (see below).
We have two main ways that papers and research projects enter the Unjournal process:
For either track, authors are invited to be involved in several ways:
Authors are informed of the process and given an opportunity to identify particular concerns, request an embargo, etc.
Evaluators can be put in touch with authors (anonymously) for clarification questions.
Authors are given a two-week window to respond to the evaluations (this response is published as well) before the evaluations are made public. They can also respond after the evaluations are released.
If you are writing a signed evaluation, you can share it or link it on your own pages. Please wait to do this until after we have given the author a chance to respond and posted the package.
Otherwise, if you are remaining anonymous, please do not disclose your connection to this report.
Going forward:
As a general principle, we hope and intend always to see that you are fairly compensated for your time and effort.
For careers and improving research: Evaluations provide metrics of quality. In the medium term, these should provide increased and accelerated career value, improving the research process. We aim to build metrics that are credibly comparable to the current "tier" of journal a paper is published in. But we aim to do this better in several ways:
Feedback and suggestions for authors: We expect that evaluators will provide feedback that is relevant to the authors, to help them make the paper better.
We still want your evaluation and ratings. Some things to consider as an evaluator in this situation.
A paper/project is not only a good to be judged on a single scale. How useful is it, and to who or what? We'd like you discuss its value in relation to previous work, it’s implications, what it suggests for research and practice, etc.
Even if the paper is great...
Would you accept it in the “top journal in economics”? If not, why not?
Would you hire someone based on this paper?
Would you fund a major intervention (as a government policymaker, major philanthropist, etc.) based on this paper alone? If not, why not
What are the most important and informative results of the paper?
Can you quantify your confidence in these 'crucial' results, and their replicability and generalizability to other settings? Can you state your probabilistic bounds (confidence or credible intervals) on the quantitative results (e.g., 80% bounds on QALYs/DALYs/or WELLBYs per $1000)
Would any other robustness checks or further work have the potential to increase your confidence (narrow your belief bounds) in this result? Which?
Do the authors make it easy to reproduce the statistical (or other) results of the paper from shared data? Could they do more in this respect?
Communication: Did you understand all of the paper? Was it easy to read? Are there any parts that could have been better explained?
Is it communicated in a way that would it be useful to policymakers? To other researchers in this field, or in the general discipline?
For several reasons... (for more discussion, see )
Paying evaluators can reduce and conflicts of interest —arguably inherent to the traditional process where reviewers work for free.
Yes, we allow evaluators to choose whether they wish to remain anonymous or "sign" their evaluations. See .
We will put your evaluation on and give it a DOI. It cannot be redacted in the sense that this initial version will remain on the internet in some format. But you can add an addendum to the document later, which we will post and link, and the DOI can be adjusted to point to the revised version.
See the FAQ as well as the .
Authors ; if we believe the work is relevant, we assign evaluators, and so on.
We that seems potentially influential, impactful, and relevant for evaluation. In some cases, we request the authors' permission before sending out the papers for evaluation. In other cases (such as where senior authors release papers in the prestigiousand CEPR series) we contact the authors and request their engagement before proceeding, but we don't ask for permission.
We may later invite you to . . .
. . . and to help us judge prizes (e.g., the ).
We may ask if you want to be involved in replication exercises (e.g., through the ).
The evaluations provide at least three types of value, helping advance several paths in our :
For readers and users: Unjournal evaluations assess the reliability and usefulness of the paper along several dimensions—and make this public, so other researchers and policymakers can
More quickly, more reliably, more transparently, and without the unproductive overhead of dealing with journals (see '')
Allowing flexible, , thus improving the research process, benefiting research careers, and hopefully improving the research itself in impactful areas.
See "what
See our .
How does The Unjournal's evaluation process work?
We generally seek (aka reviewers or referees) with research interests in your area and with complementary expertise. You, the author, can suggest areas that you want to on.
The evaluators write detailed and helpful evaluations, and submit them either "signed" or anonymously. They provide quantitative ratings on several dimensions, such as methods, relevance, and communication. They predict what journal tier the research will be published in, and what tier it should be published in. Here are the Guidelines for evaluators.
These evaluations and ratings are typically made public (see unjournal.pubpub.org), but you will have the right to respond before or after these are posted.
To consider your research we only need a link to a publicly hosted version of your work, ideally with a DOI. We will not "publish" your paper. The fact that we are handling your paper will not limit you in any way. You can submit it to any journal before, during, or after the process.
You can request a conditional embargo by emailing us at contact@unjournal.org, or via the submission form. Please explain what sort of embargo you are asking for, and why. By default, we would like Unjournal evaluations to be made public promptly. However, we may make exceptions in special circumstances, particularly for very early-career researchers.
If there is an early-career researcher on the authorship team, we may allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite.
For further details on this, and examples, see "Conditional embargoes and exceptions".
We may ask for some of the below, but these are mainly optional. We aim to make the process very light touch for authors.
A link to a non-paywalled, hosted version of your work (in ) which .
Responses to .
We may ask for a link to data and code, if possible. Note that our project is not principally about replication, and we are not insisting on this.
We also allow you to respond to evaluations, and we give your response its own DOI.
The biggest personal gains for :
Substantive feedback will help you improve your research. Substantive and useful feedback is often very hard to get, especially for young scholars. It's hard to get anyone to read your paper – we can help!
Ratings are markers of credibility for your work that could help your career. Remember, we select our research based on potential global relevance.
The chance to publicly respond to criticism and correct misunderstandings.
A connection to the EA/Global Priorities communities and the Open Science movement. This may lead to grant opportunities, open up new ambitious projects, and attract strong PhD students to your research groups.
A reputation as an early adopter and innovator in open science.
Prizes: You may win an Impactful Research Prize (pilot) which will be financial as well as reputational.
Undervalued or updated work: Your paper may have been "under-published". Perhaps there are a limited set of prestigious journals in your field. You now see ways you could improve the research. The Unjournal can help; we will consider post-publication evaluation. (See fold.)
Innovative formats: Journals typically require you to submit a LaTeX or MS Word file, and to use their fussy formats and styles. You may want to use tools like Quarto that integrate your code and data, allow you to present dynamic content, and enhance reproducibility. The Unjournal , and we can evaluate research in virtually any format.
There are risks and rewards to any activity, of course. Here we consider some risks you may weigh against the benefits mentioned above.
Exclusivity
Public negative feedback
, and perhaps they might enforce these more strongly if they fear competition from The Unjournal.
However, The Unjournal is not exclusive. Having your paper reviewed and evaluated in The Unjournal will not limit your options; you can still submit your work to traditional journals.
Our evaluations are public. While there has been some movement towards open review, this is still not standard. Typically when you submit your paper, reviews are private. With The Unjournal, you might get public negative evaluations.
We think this is an acceptable risk. Most academics expect that opinions will differ about a piece of work, and everyone has received negative reviews. Thus, getting public feedback — in The Unjournal or elsewhere — should not particularly harm you or your research project.
Nonetheless, we are planning some exceptions for early-career researchers (see #can-i-ask-for-evaluations-to-be-private).
Unjournal evaluations should be seen as signals of research quality. Like all such signals, they are noisy. But submitting to The Unjournal shows you are confident in your work, and not afraid of public feedback.
Within our "Direct evaluation" track, The Unjournal directly chooses papers (from prominent archives, well-established researchers, etc.) to evaluate. We don't request authors' permission here.
As you can see in our evaluation workflow, on this track, we engage with authors at (at least) two points:
Informing the authors that the evaluation will take place, requesting they engage, and giving them the opportunity to request a conditional embargo or specific types of feedback.
Of particular interest: are we looking at the most recent version of the paper/project, or is there a further revised version we should be considering instead?
After the evaluations have been completed, the authors are given two weeks to respond, and have their response posted along with our 'package'. (Authors can also respond after we have posted the evaluations, and we will put their response in the same 'package', with a DOI etc.)
Once we receive unsolicited work from an author or authors, we keep it in our database and have our team decide on prioritization. If your paper is prioritized for evaluation, The Unjournal will notify you.
At present, we do not have a system to fully share the status of author submissions with authors. We hope to have one in place in the near future.
You can still submit your work to any traditional journal.
The Unjournal aims to evaluate the most recent version of a paper. We reach out to authors to ensure we have the latest version at the start of the evaluation process.
If substantial updates are made to a paper during the evaluation process, authors are encouraged to share the updated version. We then inform our evaluators and ask if they wish to revise their comments.
If the evaluators can't or don't respond, we will note this and still link the newest version.
Authors are encouraged to respond to evaluations by highlighting major revisions made to their paper, especially those relevant to the evaluators' critiques. If authors are not ready to respond to evaluations, we can post a placeholder response indicating that responses and/or a revised version of the paper are forthcoming.
Re-evaluation: If authors and evaluators are willing to engage, The Unjournal is open to re-evaluating a revised version of a paper after publishing the evaluations of the initial version.
We share evaluations with the authors and give them a chance to respond before we make the evaluations public (and again afterward, at any point). We add these to our evaluation packages on PubPub. Evaluation manager's (public) reports and our further communications incorporate the paper, the evaluations, and the authors' responses.
Authors' responses could bring several benefits...
Personally: a chance to correct misconceptions and explain their approach and planned steps. If you spot any clear errors or omissions, we give evaluators a chance to adjust their reports in response. The authors response can also help others (including the evaluators, as well as journal referees and grant funders) to have a more accurate and positive view of the research
For research users, to get an informed balanced perspective on how to judge the work
For other researchers, to better understand the methodological issues and approaches. This can serve to start a public dialogue and discussion to help build the field and research agenda. Ultimately, we aim to facilitate a back-and-forth between authors, evaluators, and others.
Examples: We've received several detailed and informative author responses, such as:
Evaluations may raise substantive doubts and questions, and make some specific suggestions, and ask about (e.g.) data, context, or assumptions. There's no need to respond to every evaluator point; only you have something substantive: clarifying doubts, explaining the justification for your particular choices, and giving your thoughts on the suggestions (which will you incorporate, or not, and why?).
A well-written author response would (ideally) have a clear narrative and group responses into themes.
Try to have a positive tone (no personal attacks etc.) but avoid too much formality, politeness, or flattery. Revise-and-resubmit responses at standard journals sometimes begin each paragraph with "thank you for this excellent suggestion". Feel free to skip this; we want to focus on the substance.