31 Aug 2023: Our present approach is a "working solution" involving some ad-hoc and intuitive choices. We are re-evaluating the metrics we are asking for as well as the interface and framing. We are gathering some discussion in this linked Gdoc, incorporating feedback from our pilot evaluators and authors. We're also talking to people with expertise as well as considering past practice and other ongoing initiatives. We plan to consolidate that discussion and our consensus and/or conclusions into the present (Gitbook) site.
Why numerical ratings?
Ultimately, we're trying to replace the question of "what tier of journal did a paper get into?" with "how highly was the paper rated?" We believe this is a more valuable metric. It can be more fine-grained. It should be less prone to gaming. It aims to reduce randomness in the process, through things like 'the availability of journal space in a particular field'. See our discussion of Reshaping academic evaluation: beyond the binary... .
To get to this point, we need to have academia and stakeholders see our evaluations as meaningful. We want the evaluations to begin to have some value that is measurable in the way “publication in the AER” is seen to have value.
While there are some ongoing efforts towards journal-independent evaluation, these . Typically, they either have simple tick-boxes (like "this paper used correct statistical methods: yes/no") or they enable descriptive evaluation without an overall rating.
As we are not a journal, and we don’t accept or reject research, we need another way of assigning value. We are working to determine the best way of doing this through quantitative ratings. We hope to be able to benchmark our evaluations to "traditional" publication outcomes. Thus, we think it is important to ask for both an overall quality rating and a journal ranking tier prediction.
Why these categories?
In addition to the overall assessment, we think it will be valuable to have the papers rated according to several categories. This could be particularly helpful to practitioners who may care about some concerns more than others. It also can be useful to future researchers who might want to focus on reading papers with particular strengths. It could be useful in meta-analyses, as certain characteristics of papers could be weighed more heavily. We think the use of categories might also be useful to authors and evaluators themselves. It can help them get a sense of what we think research priorities should be, and thus help them consider an overall rating.
However, these ideas have been largely ad-hoc and based on the impressions of our management team (a particular set of mainly economists and psychologists). The process is still being developed. Any feedback you have is welcome. For example,are we overemphasizing certain aspects? Are we excluding some important categories?
We are also researching other frameworks, templates, and past practice; we hope to draw from validated, theoretically grounded projects such asRepliCATS.
Why ask for credible intervals?
In eliciting expert judgment, it is helpful to differentiate the level of confidence in predictions and recommendations. We want to know not only what you believe, but how strongly held your beliefs are. If you are less certain in one area, we should weigh the information you provide less heavily in updating our beliefs. This may also be particularly useful for practitioners.
Obviously, there are challenges to any approach. Even experts in a quantitative field may struggle to convey their own uncertainty. They may also be inherently "poorly calibrated" (see discussions and tools for calibration training). Some people may often be "confidently wrong." They might state very narrow "credible intervals", when the truth—where measurable—routinely falls outside these boundaries. People with greater discrimination may sometimes be underconfident. One would want to consider and
As a side benefit, this may be interesting for research , particularly as The Unjournal grows. We see 'quantifying one's own uncertainty' as a good exercise for academics (and everyone) to engage in.
"Weightings" for each rating category (removed for now)
Weightings for each ratings category (removed for now)
2 Oct 2023 -- We previously suggested 'weightings' for individual ratings, along with a note
We give "suggested weights" as an indication of our priorities and a suggestion for how you might average these together into an overall assessment; but please use your own judgment.
We included these weightings for several reasons:
People are found [reference needed] do a more careful job at prediction (and thus perhaps at overall rating too) if the outcome of interest is built up from components that are each judged separately.
We wanted to make the overall rating better defined and thus more useful to outsiders and comparable across raters
Emphasizing what we think is important (in particular, methodological reliability)
We didn't want evaluators to think we wanted them to weigh each category equally … some are clearly more important
However, we decided to remove these weightings because:
Reduce clutter in an already overwhelming form and guidance doc. ‘More numbers’ can be particularly overwhelming
These weights were ad-hoc, and they may suggest we have a more grounded ‘model of value’ than we already do. (And there is also some overlap in our categories anyways, something we are working on addressing.)
Some people interpreted what we intended incorrectly (e.g., they thought we were saying ‘relevance to global priorities’ is not an important thing)
Adjustments to metrics and guidelines/previous presentations
Oct 2023 update - removed "weightings"
We have removed suggested weightings for each of these categories. We discuss the rationale at some length here.
Evaluators working before October 2023 saw a previous version of the table, which you can see HERE.
We previously gave evaluators two options for expressing their confidence in each rating:
Either:
The 90% Confidence/Credible Interval (CI) input you see below (now a 'slider' in PubPub V7) or
A five-point 'Likert style' measure of confidence, which we described qualitatively and explained how we would convert it into CIs when we report aggregations.
To make this process less confusing, to encourage careful quantification of uncertainty, and to enable better-justified aggregation of expert judgment, we are de-emphasizing the latter measure.
Still, to accommodate those who may not be familiar with or comfortable stating "90% CIs on their own beliefs" we offer further explanations, and we are providing tools to help evaluators construct these. As a fallback, we will still allow evaluators to give the 1-5 confidence measure, noting the correspondence to CIs, but we discourage this somewhat.
The previous guidelines can be seen here; these may be useful in considering evaluations provided pre-2024.
Pre-October 2023 'ratings with weights' table, provided for reference (no longer in use)
Category (importance)
Sugg. Wgt.*
Rating (0-100)
90% CI
Confidence (alternative to CI)
44
39, 52
5
50
47, 54
5
51
45, 55
4
20
10, 35
3
60
40, 70
2
35
30,46
0**
30
21,65
We had included the note:
We give the previous weighting scheme in a fold below for reference, particularly for those reading evaluations done before October 2023.
As well as:
Suggested weighting: 0.
Elsewhere in that page we had noted:
As noted above, we give suggested weights (0–5) to suggest the importance of each category rating to your overall assessment, given The Unjournal's priorities.
Pre-2024 ratings and uncertainty elicitation, provided for reference (no longer in use)
Category (importance)
Rating (0-100)
90% CI
Confidence (alternative to CI)
44
39, 52
50
47, 54
51
45, 55
20
10, 35
60
40, 70
35
30,46
30
21,65
[FROM PREVIOUS GUIDELINES:]
You may feel comfortable giving your "90% confidence interval," or you may prefer to give a "descriptive rating" of your confidence (from "extremely confident" to "not confident").
4 = Very confident: 90% confidence interval +/– 8 points or less
3 = Somewhat confident: 90% confidence interval +/– 15 points or less
2 = Not very confident: 90% confidence interval, +/– 25 points or less
1 = Not confident: 90% confidence interval +/– 25 points
Previous 'descriptions of ratings intervals'
[Previous guidelines]: The description folded below focuses on the "Overall Assessment." Please try to use a similar scale when evaluating the category metrics.
Top ratings (90–100)
95–100: Among the highest quality and most important work you have ever read.
90–100: This work represents a major achievement, making substantial contributions to the field and practice. Such work would/should be weighed very heavily by tenure and promotion committees, and grantmakers.
For example:
Most work in this area in the next ten years will be influenced by this paper.
This paper is substantially more rigorous or more insightful than existing work in this area in a way that matters for research and practice.
The work makes a major, perhaps decisive contribution to a case for (or against) a policy or philanthropic intervention.
Near-top (75–89) (*)
This work represents a strong and substantial achievement. It is highly rigorous, relevant, and well-communicated, up to the standards of the strongest work in this area (say, the standards of the top 5% of committed researchers in this field). Such work would/should not be decisive in a tenure/promotion/grant decision alone, but it should make a very solid contribution to such a case.
Middle ratings (40–59, 60–74) (*)
: A very strong, solid, and relevant piece of work. It may have minor flaws or limitations, but overall it is very high-quality, meeting the standards of well-respected research professionals in this field.
40–59.9: A useful contribution, with major strengths, but also some important flaws or limitations.
Low ratings (5–19, 20–39) (*)
20–39.9: Some interesting and useful points and some reasonable approaches, but only marginally so. Important flaws and limitations. Would need substantial refocus or changes of direction and/or methods in order to be a useful part of the research and policy discussion.
5–19.9: Among the lowest quality papers; not making any substantial contribution and containing fatal flaws. The paper may fundamentally address an issue that is not defined or obviously not relevant, or the content may be substantially outside of the authors’ field of expertise.
0–4: Illegible, fraudulent, or plagiarized. Please flag fraud, and notify us and the relevant authorities.
(*) 20 Mar 2023: We adjusted these ratings to avoid overlap
The previous categories were 0–5, 5–20, 20–40, 40–60, 60–75, 75–90, and 90–100. Some evaluators found the overlap in this definition confusing.
This page explains the value of the metrics we are seeking from evaluators.
The Calibrate Your Judgment app from Clearer Thinking is fairly helpful and fun for practicing and checking how good you are at expressing your uncertainty. It requires creating account, but that doesn't take long. The 'Confidence Intervals' training seems particularly relevant for our purposes.
