Comment on page
Global priorities: Theory of Change
Our 'Theory of Change' paths are shown above; we highlight what is arguably the most 'direct' path in yellow. All of these begin with our setting up, funding, communicating, and incentivizing participation in a strong, open, efficient research evaluation system (in green, at top). These all lead to impactful research being more in-depth, more reliable, more accessible, and more useful, better informing decision-makers, and leading to better decisions and outcomes (in green, at bottom).
Highlighting some of the key paths:
- 1.(Yellow) Faster and better feedback on impactful research improves this work, and better informs policymakers and philanthropists (Yellow path)
- 2.(Blue) Our processes and incentives will foster ties between mainstream/prominent/academic/policy researchers and global-priorities/EA-focused researchers. This will improve the rigor, credibility, exposure, and influence of previously ‘EA niche’ work, while helping mainstream researchers better understand and incorporate ideas, principles and methods from the EA and rationalist research communities (counterfactual impact, cause-neutrality, reasoning transparency, etc.)
- 1.This process will also nudge mainstream academics towards focusing on impact and global priorities, and towards making their research and outputs more accessible and useable.
- 3.The Unjournal’s more efficient, open, and flexible processes will become attractive to academics and stakeholders. As we become better at ‘predicting publication outcomes’, we will become a replacement for traditional processes, improving research overall (some of which will be highly impactful research).
Rigorous quantitative and empirical research in economics, business, public policy, and social science has the potential to improve our decision-making and enable a flourishing future. This can be seen in the research frameworks proposed by 80,000 hours, Open Philanthropy, and The Global Priorities Institute (see What is EA/gp relevant research?). This research is routinely used by effective altruists working on global priorities or existential risk mitigation. It informs both philanthropic decisions (e.g., those influenced by GiveWell's Cost-Effectiveness Analyses, whose inputs are largely based on academic research) and national public policy
However, the academic publication process is notoriously slow; for example, in economics it routinely takes 2-6 years between the first presentation of a research paper and the eventual publication in a peer-reviewed journal. Recent reforms have sped up parts of the process by encouraging researchers to put working papers/preprints online.
However, working papers and preprints often receive at most only a cursory check before publication, and it is up to the reader to judge quality for themselves. Decision-makers and other researchers rely on peer review to judge the work’s credibility. This part remains slow and inefficient. Furthermore, it provides very noisy signals: a paper is typically judged by the ‘prestige of the journal it lands in’ (perhaps after an intricate odyssey across journals), but it is hard to know why it ended up there. Publication success is seen to depend on personal connections, cleverness, strategic submission strategies, good presentation skills, and relevance to the discipline’s methods and theory. These factors are largely irrelevant to whether and how philanthropists and policymakers should consider and act on a paper’s claimed findings. Reviews are kept secret; the public never learns why a paper was deemed worthy of a journal, nor what its strengths and weaknesses were.
We believe that disseminating research sooner – along with measures of its credibility– is better.
We also believe that publicly evaluating its quality before (and in addition to) journal publication will add substantial additional value, providing:
- 1.A quality assessment (by experts in the field) that can be read alongside the preprint by decision makers or other researchers, helping these users weigh its strengths and weaknesses, and interpreting its implications.
- 2.Faster feedback to authors, focused on improving the rigor and impact of the work.
Various initiatives in the life sciences are already begun reviewing preprints. While economics took the lead in sharing working papers, public evaluation of economics, business, and social science research is rare. The Unjournal is the first initiative to publicly evaluate rapidly-disseminated work from these fields. Our specific priority: research relevant to global priorities
The Unjournal will encourage and incentivize substantive and helpful feedback and careful quantitative evaluation. We will publish these evaluations in a carefully curated space, and clearly aggregate and communicate this output.
This will help us achieve our focal, most tangible ‘theory of change’ pathway (mapped in our ‘Plan for Impact’.)
- better (faster, public, more thorough, more efficient, quantified, and impact-minded) evaluation of pivotal research
- makes this research better (both the evaluated work and adjacent work), encourages more such work
- and makes it easier for decision-makers to evaluate and use it, leading to better decisions and better outcomes,
- reducing X-risk and contributing to long-term survival and flourishing.
The Unjournal’s open feedback should also be valuable to the researchers themselves and their research community, catalyzing progress. As the Unjournal Evaluation becomes a valuable outcome in itself, researchers can spend less time ‘gaming the journal system’. Shared public evaluation will provide an important window to other researchers, helping them better understand the relevant cutting-edge concerns. The Unjournal will permit research to be submitted in a wider variety of useful formats (e.g., dynamic documents and notebooks rather than ‘frozen pdfs’), enabling more useful, replicable content and less time spent formatting papers for particular journals. We will also allow researchers to improve their work ‘in situ’ and gain updated evaluations, rather than having to spin-off new papers. This will make the literature more clear and less cluttered.
Some of the paths will take longer than others; in particular, it will be hard to get academia to change, particularly because of entrenched systems and a collective action problem. We discuss how we hope to overcome this HERE. In particular:
We can provide leadership and take risks that academics won’t take themselves.
- Bringing in new interests, external funding, and incentives can change the fundamental incentive structure.
- We can play a long game and build our processes and track record, while we wait for academia to directly incorporate journal-independent evaluations int0 their reward systems. Meanwhile, our work and output will be highly useful to EA/global priorities longtermist researchers and decision-makers. as part of their metrics and reward systems.
- The Unjournal’s more efficient, open, and flexible processes will become attractive to academics and stakeholders. As we become better at ‘predicting publication outcomes’, we will become a replacement for traditional processes, improving research overall (some of which will be highly impactful research).
- This process will also nudge mainstream academics towards focusing on impact and global priorities, and towards making their research and outputs more accessible and useable.
.
Last modified 6mo ago