Comment on page
An introduction to The Unjournal
Journal-independent evaluation of impactful research
We are a team of researchers, practitioners, and open science advocates building a better system for evaluating rigorous, impactful research.
We are not a journal!
The Unjournal will not "publish papers in a journal"; instead, we coordinate and finance the public evaluation of hosted papers and dynamically-presented research projects. We are working independently of traditional academic journals to build an open platform and a sustainable system for feedback, ratings, and assessment.
Our initial focus is quantitative work that informs global priorities, especially in economics, policy, and other social sciences. We will encourage better research by making it easier for researchers to get feedback and credible ratings on their work.
Why The Unjournal?
Peer review is great, but conventional academic publication processes are wasteful, slow, and rent-extracting. They discourage innovation and prompt researchers to focus more on "gaming the system" than on the quality of their research. We will provide an immediate alternative and at the same time offer a bridge to a more efficient, informative, useful, and transparent research evaluation system.
- 1.Research submission/identification and selection: We'll identify, solicit, and select relevant research work to be hosted on any open platform in any format that can gain a time-stamped DOI. Authors are encouraged to present their work in the ways they find most comprehensive and understandable. We'll support the use of dynamic documents and other formats that foster replicability and open science (see: the benefits of dynamic docs).Direct evaluation of high-impact work in the high-profile NBER working papers series will also be included (see: direct evaluation track).
- 2.Paid evaluators (AKA "reviewers"): We will compensate evaluators (essentially, reviewers) for providing thorough feedback on this work. (Read more: Why do we pay?)
- 3.Eliciting quantifiable and comparable metrics: We aim to establish and generate credible measures of research quality and usefulness. We intend to benchmark these against traditional previous measures (journal tiers, etc.) and assess the reliability, consistency, and predictive power of these measures. (Read more: Why quantitative metrics?)
- 4.Public evaluation: Reviews will typically be made public, including potential author responses. This will facilitate dialogue and enhance understanding.
- 5.Linking, not publishing: Our process will not be "exclusive." Authors can submit their work to a journal at any time. This approach also allows us to benchmark our evaluations against traditional publication outcomes.
- 6.Financial prizes: We will award financial prizes, paired with public presentations, to works judged to be the strongest.
- 7.Transparency: We will aim for maximum transparency in our processes and judgments.
For example, this proposal is closely related to Life's "Publish, Review, Curate" model; see their updated (Oct 2022) model here. However, we cover a different research focus and make some different choices, discussed below.
Below, we also discuss other Parallel/partner initiatives and resources, many of whom we hope to work with. However, we think we are the only group funded to do this in this particular research area/focus. We are also taking a different approach to previous efforts, including funding evaluation (see Why pay evaluators (reviewers)?) and asking for quantified ratings and predictions (see Guidelines for evaluators).
Our current funding comes from:
Survival and Flourishing Fund (successful)
ACX/LTFF grant proposal (as submitted, successful) grant (ACX passed it to the Long Term Future Fund, who awarded it). This funding was extended through mid-2023.
We have submitted some other grant applications; e.g., see our unsuccessful FTX application here; other grant applications are linked below. We are sharing these in the spirit of transparency.
Academics and funders have complained about this stuff for years and continue to do so every day on social media . . . and we suspect our critiques of the traditional review and publication process will resonate with readers.
So why haven't academia and the research community been able to move to something new? There is a difficult collective action problem. Individual researchers and universities find it risky to move unilaterally. But we believe we have a good chance of finally changing this model and moving to a better equilibrium, because we will:
- Take risks: Many members of The Unjournal management are not traditional academics; we can stick our necks out. We are also bringing on board established senior academics who are less professionally vulnerable.
- Bring in new interests, external funding, and incentives: There are a range of well-funded and powerful organizations—such as the Sloan Foundation and Open Philanthropy—with a strong inherent interest in high-impact research being reliable, robust, and reasoning-transparent. This support can fundamentally shift existing incentive structures.
- Allow less risky "bridging steps": As noted above, The Unjournal allows researchers also to submit their work to traditional journals. In fact, this will provide a benchmark to help build our quantitative ratings and demonstrate their value.
- Communicate with researchers and stakeholders to make our processes easy, clear, and useful to them.
- Make our output useful: It may take years for university departments and grant funders to incorporate journal-independent evaluations as part of their metrics and reward systems. The Unjournal can be somewhat patient: our evaluation, rating, feedback, and communication will provide a valuable service to authors, policymakers, and other researchers in the meantime.
- Leverage new technology: A new set of open-access tools (such as those funded by Sloan Scholarly Communications) makes what we are trying to do easier, and makes formats other than static PDFs more useful every day.
- Reward early adopters with prizes and recognition: We can replace "fear of standing out" with "fear of missing out." In particular, authors and research institutions that commit to publicly engaging with evaluations and critiques of their work should be commended and rewarded. And we intend to do this.
This GitBook serves as a platform to organize our ideas and resources and track our progress towards The Unjournal's dual objectives:
- 1.Making "peer evaluation and rating" of open projects into a standard high-status outcome in academia and research, specifically within economics and social sciences. This stands in contrast to the conventional binary choice of accepting or rejecting papers to be published as PDFs and other static formats.
- 2.Building a cohesive and efficient system for publishing, accruing credibility, and eliciting feedback for research aligned with effective altruism and global priorities. Our ultimate aim is to make rigorous research more impactful, and impactful research more rigorous.
Discussion space: I've set up a post-grant "'Unjournal: Action plan discussion space" here. 25 Jun 2022 update: I have not kept the above discussion space fully updated.
Please let me know if you want edit/comment access to the present Gitbook.
Please do weigh in, all suggestions and comments will be credited... See also Unjournal: public-facing FAQ in progress.
- 1.We target these areas (1) because of our current management team's expertise and (2) because these seem particularly in need of The Unjournal's approach. However, we are open to expanding and branching out.
- 2.We are considering future outcomes like replication and citations.
- 3.We will also consider funding later rounds of review or evaluations of improved and expanded versions of previously-evaluated work.
- 4.Please let me know if you want edit or comment access to the present Gitbook. Please do weigh in, all suggestions and comments will be credited. See also Unjournal: public-facing FAQ in progress.
Last modified 8h ago