> For the complete documentation index, see [llms.txt](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/policies-projects-evaluation-workflow/evaluation/guidelines-for-evaluators/ai-llm-policy.md). # AI/LLM use in evaluations *This is an evolving policy; we have consulted with our Management Team but have not formally confirmed all aspects.* ## Framework Our goal is relevant, insightful, and "correct[^1]" evaluations that fairly assess research credibility and value as well as the credibility of specific claims, help researchers improve their work, and help practitioners and other researchers use it appropriately. We want evaluators (and our process) to have access to all tools that facilitate this. At the same time, we are mindful of other concerns, including: * continuing to cultivate and demonstrate the value of our evaluation process; * encouraging researchers and practitioners to continue to work with us; * capturing credible and independent sources of judgment and reasoning; * being transparent with readers of our evaluations; * [communicating our work in useful ways with stakeholders;](#user-content-fn-2)[^2] * supporting global priorities for the wellbeing of humans, animals, and society; * keeping AI's role in research aligned with human and social interest, preserving human oversight and control, and understanding AI capabilities and alignment in these areas. ## AI audit and evaluation tools we will provide *Before the evaluations:* We will recommend facilitate the use of consistency and 'bug-checking' tools, such as RegCheck (a preregistration checker); we aim to assess the available tools, including those listed [here](https://daaronr.github.io/unjournal_tools_interfaces/evaluation_form/tools.html). *After evaluators have submitted their initial evaluations*, we aim to provide (and later share publicly): 1. State of the art (flagship model, bespoke agent-based tools, etc.) AI evaluations of the paper 2. AI consistency checks of the human evaluation against the paper Evaluators are advised adjust their evaluations (and perhaps adjust their ratings) if they are confident the LLM has identified an important misunderstanding or omissionin their evaluation, and to note any such adjustments. If resources and bandwidth allows, they will also be encouraged to provide feedback on these AI evaluations, which will serve as a valuable benchmarking output in itself. *Consider*: * Shall we publicly report both the initial and revised ratings? * Should we encourage evaluators *not* to update their ratings in response to seeing AI ratings or general characterizations; only to specific identified misunderstandings/omissions. ## What evaluators may use Evaluators may use AI tools selectively. Appropriate uses include looking up or clarifying content in the paper (e.g. using NotebookLM as a reference tool), or running extensive checks that are not feasible to do by hand: in which case the AI-assisted section should appear separately (in a dedicated section, link, or footnote) and the evaluators should include at least a hand-checked sample. *Consider:* Where the evaluator lacks expertise in a particular subject, shall we recommend against using AI tools to complement their own knowledge or expertise, except where this involves a simple 'lookup or clarification'? Rationale — we aim to provide a corpus human expertise and judgment, rather than to propagate existing AI model content ('contamination'). While, in principle, experts could augment their understanding by using AI to learn about a specific technique and improve their human advising, this again seems likely to lead to 'contamination'. As a compromise, evaluators could include small \~appendix sections specifying the specific queries they performed, and the insights gined from these, but these should not form the main substance of the report. Evaluators are encouraged *not* to use AI tools for overall evaluations or for the ratings and predictions components, except as described above under what we provide. ## Transparency requirements Evaluators must report how AI tools were used by: 1. Providing links to their AI chats or explanations (where possible) 2. Explicitly identifying any sections of text that were generated by LLMs ## Human work requirement Evaluators are expected to put in at least 8 hours of human work, over and above the latent processing time of any LLM tools. ## Standing by your evaluation Evaluators must stand by all content and language in their report as their own judgment. They must independently verify and carefully consider any points raised by AI tools. ## Related evidence and context *13 Jul 2026 —* this bears updating. One internal update — our own [experimentation with late-2025 tools](https://llm-uj-research-eval.netlify.app/appendix_llm_traces) suggested these provided substantial value, and made points that often overlapped with our human evaluators (even though these evaluations should have been out of the model context window). *Previous note;* Emerging evidence points toward a complementary role for AI in peer review processes like ours. Biswas et al. (2025), [AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot](https://arxiv.org/abs/2604.13940)(arXiv:2604.13940), report on a large-scale survey of AAAI-26 authors and program committee members and found "participants not only found AI reviews useful, but actually preferred them to human reviews on key dimensions such as technical accuracy and research suggestions". The system processed all 22,977 full-paper submissions in under a day using frontier models with tool use and safeguards. This offers promising evidence for AI-assisted peer review at scale. But we are being cautious — we consider we use AI checks as a complement and a supplement to human evaluation, preserving independent expert judgment while adding an additional consistency check and source of suggestions. *Also relevant: Johnny Coates'* [*"Best Practices for preprint peer review services in the use of AI"*](https://docs.google.com/document/d/17Fz4opzft7mfvVMbnU8bnwXNabWnuw8F8YWS3whLMqM/edit?tab=t.0)*.* [^1]: Naturally there are different interpretations of methodology, context, relevance, etc. By *correct* we mean accurately reflecting the paper's content, embodying expertise, making careful calibrated statements, factually accurate, consistent with a reasonaable methodological approach, etc. [^2]: 13 Jul 2026 — added the latter three bullet points to emphasize the risks of deferring too much to AI, and the benefits of keeping a human-in-the loop and checking the capabilities and alignment of AI work. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://globalimpact.gitbook.io/the-unjournal-project-and-communication-space/policies-projects-evaluation-workflow/evaluation/guidelines-for-evaluators/ai-llm-policy.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.