Comment on page
For researchers and authors
This section is specifically for researchers or authors considering submitting their work. We address the benefits of submission and discuss how we are mitigating or can counter potential risks.
We generally seek two evaluators˟ (aka reviewers/referees') with research interests in your area, and with complementary expertise. You will have the opportunity to suggest particular areas in which you are interested in receiving feedback and suggestions.˟ The evaluators will write detailed and helpful evaluations, and submit them either "signed" or anonymously. We will also ask the evaluators to give some forms of quantitative "rating" feedback, along with a few overall metrics and predictions about what tier and type of traditional journal this work would likely be published in, and where it should be published (see note). All of these evaluations and ratings will typically be made public, but you will have the right to respond before or after these are posted.
There will be a range of benefits for authors, as discussed below.
We will only need a link to a publicly hosted, DOI'd version of your work. We will not "publish" your paper, and the fact that we are handling your paper will not limit you in any way. We are fine with you submitting it to any journal before, during, or after the process. (See note.)
By default, we would like Unjournal evaluations to be made public. However, in special circumstances, and particularly for very early-career researchers, we may make exceptions.
If there is an early-career researcher on the authorship team, we will allow authors to "embargo" the publication of the evaluation until a later date. Evaluators (referees) will be informed of this. This date can be contingent, but it should not be indefinite. For example, we might grant an embargo that lasts until after a PhD/postdoc’s upcoming job-market or until after publication in a mainstream journal, with a hard maximum of 14 months. (Of course, embargoes can be ended early at the request of the authors)
In very special circumstances we may consider granting a "conditional indefinite embargo."
If we consider your work for public evaluation, we may ask for some of the below, but these are mainly optional. We will aim to make this very light touch for authors.
- 1.A link to a non-paywalled, hosted version of your work (which could be in any format (*)— PDFs are not necessary) which can be given a DOI.
- 2.Responses to clarification questions from our evaluators.
- 3.The evaluations and ratings will be posted publicly (and themselves given a DOI). We will allow you the right to respond before any of this is posted publicly, and your response will also be given a DOI and indexed..
- 4.We are considering asking for a link to data and code, if possible. We will help you to make it accessible. Note our project is not principally about replication, and we are not insisting on this, nor able to provide this aid systematically at this point in time.
The biggest personal gains for authors are:
- 1.Substantive feedback will help you improve your work (both the work being reviewed and future work). Substantive and useful feedback is often very hard to get, especially for young scholars.
- 2.Ratings = Markers of credibility for your work that could help your career advancement, at least at the margin, and could in the future help a great deal.
- 3.The opportunity to publicly respond to critiques and correct misunderstandings.
- 4.An additional opportunity for evaluation of your work with an emphasis on impact. (See fold below).
- 5.Visibility and attention to your work, which may lead to additional citations.
- 6.A connection to the EA/Global Priorities communities and the Open Science movement. This may lead to grant opportunities, opening up new ambitious projects and attracting strong PhD students to your research groups.
- 7.A reputation as an "early adopter and innovator" in open science.
- 8.
- 9.Future opportunities: Being part of our system makes you more likely to be hired as a paid reviewer or editorial manager.
Your work may be "under-published"; perhaps you were in a hurry and submitted it to a "safe" but low-ranked journal.
It may have been evaluated by a journal in one field, but it needs feedback and credibility from other fields (e.g., theory vs. empirics, etc.).
You may have substantially improved and extended the work since its publication.
You may have "used up" the good journals in your field, or otherwise fallen through the cracks; perhaps the paper is very impactful and the empirics are strong, but the field doesn't see it as innovative or clever. You need another opportunity.
There are risks and rewards to any activity of course, and we cannot predict the consequences of any given action. Here we consider some risks you may weigh against the benefits mentioned above.
- 1.Exclusivity
- 2.Public negative/neutral feedback
- 3.Backlash against innovation or "seeming weird"
- 4.Potential of the review process having a downward bias
Some traditional journals might have restrictions on the public sharing of your work, and perhaps they might enforce these more strongly if they fear competition from The Unjournal.
- The Unjournal is not exclusive. Having your paper reviewed and evaluated in The Unjournal will not limit your options; you can still submit your work to traditional journals.
- Following a traditional publishing route can also be risky: the world may change, and you may be seen as retrograde.
- More importantly, publishing (especially in Economics) is slow and has highly variable outcomes. You typically wait months for a response—sometimes more than a year; may go through many cycles of rejection, revisions, and resubmission; and may ultimately end up with an outcome (rejection) that has zero value on your CV.
- With The Unjournal your work will always be given feedback, an evaluation, and a public rating. And you can continue to improve it for future evaluation.
Our plan is to make reviews, feedback, and evaluations public. While there has been some movement towards open review, this is still not the standard mode. Typically when you submit your paper and it is rejected, you, your co-authors, the editor, and the people who wrote the reviews are the only people who actually know or see this.
With The Unjournal, you might get feedback and evaluation that is sometimes negative, and this may be made public. Is this an acceptable risk? I think so. Most academics expect that there will be many differing opinions about a piece of work, and everyone has received negative reviews throughout their career.
Suppose instead, you go the standard route, put out a working paper, and submit your paper to a sequence of journals. Suppose you present it at many seminars and finally get it published—let's say three years later (not uncommon in Economics), in a middle-status field journal. Will people think that your paper was perfect and not criticized at all? Of course not. If it had been the perfect paper it would have been immediately accepted at a "top journal." What's more, the people in your field would have been the reviewers, attended seminars, and heard all the critiques of the paper.
Thus, getting public feedback that is mixed—in The Unjournal (or anywhere)—should not particularly harm you or your research project.
And this should be weighed against the benefits of having your work in The Unjournal.
- Your willingness to engage with this process shows that you have confidence in your work and are willing to back it in public.
- The Unjournal will give authors a place to publicly respond to critiques, improve their work, and show how it has improved. Through this association, you may also have a chance for a second evaluation of the same project after iterations and improvements.
These are all major advantages over the traditional journal route, where you essentially start over with each new submission, and new referees may overlook the same point and have the same misunderstandings each time.
Nonetheless, we are planning some exceptions for early-career researchers, in some cases not publicly revealing negative evaluations of work. at least for a defined period, (and informing referees of this in advance).
Although the process is unusual, The Unjournal is not a weird concept: we are providing evaluation and feedback on research, largely with standard metrics. We are simply not tying it to journals.
Unjournal evaluations should be seen as signals of research quality. Like all such signals, they are noisy. But authors may worry a bad signal will hurt a lot, and a good signal will only help a little.
Arguments against this: Unbiased informational signals won't systematically lead to updates in one direction. For fancy and formal ways of saying this and related concepts, see Martingales, Conservation of Expected Evidence, Rational Expectations, and the Law of Iterated Expectations.
In other words, people will take your transparency into account. If a reviewer evaluates a paper without much information on how others rate it, they might suspect that there have been some substantial and important criticisms and that the paper has flaws. This is particularly the case if it has been around a while, they are submitting to a second-tier journal, etc.
On the other hand, if I see that the author has submitted it to a transparent assessment service like The Unjournal, particularly if the author has submitted a lot of his or her work to this sort of public scrutiny, I will update positively, all else being equal. The author has demonstrated they are not afraid of public feedback and openness. I will consider the public critiques, of course, but I will take into account that this paper has been held up to more scrutiny than other work I might have seen.
Of course, one cannot rule out that a negative evaluation will harm a paper's reputation, or its publication chances in any particular instance. I just don't expect this to occur systematically.
If getting any public feedback was so damaging, why would academics be eager to present their work at seminars and conferences?
- 1.
Last modified 4d ago