AI/LLM use in evaluations

The Unjournal's working policy on the use of AI and large language model (LLM) tools in the evaluation process, effective April 2026.

This is an evolving policy; we have consulted with our Management Team but have not formally confirmed all aspects.

Framework

Our goal is relevant, insightful, and "correct" evaluations that fairly assess research credibility and value as well as the credibility of specific claims, help researchers improve their work, and help practitioners and other researchers use it appropriately. We want evaluators (and our process) to have access to all tools that facilitate this.

At the same time, we are mindful of other concerns, including:

• continuing to cultivate and demonstrate the value of our evaluation process;
• encouraging researchers and practitioners to continue to work with us;
• capturing credible and independent sources of judgment and reasoning;
• being transparent with readers of our evaluations;
• communicating our work in useful ways with stakeholders;
• supporting global priorities for the wellbeing of humans, animals, and society;
• keeping AI's role in research aligned with human and social interest, preserving human oversight and control, and understanding AI capabilities and alignment in these areas.

AI audit and evaluation tools we will provide

Before the evaluations: We will recommend facilitate the use of consistency and 'bug-checking' tools, such as RegCheck (a preregistration checker); we aim to assess the available tools, including those listed here.

After evaluators have submitted their initial evaluations, we aim to provide (and later share publicly):

1. State of the art (flagship model, bespoke agent-based tools, etc.) AI evaluations of the paper
2. AI consistency checks of the human evaluation against the paper

Evaluators are advised adjust their evaluations (and perhaps adjust their ratings) if they are confident the LLM has identified an important misunderstanding or omissionin their evaluation, and to note any such adjustments. If resources and bandwidth allows, they will also be encouraged to provide feedback on these AI evaluations, which will serve as a valuable benchmarking output in itself.

Consider:

• Shall we publicly report both the initial and revised ratings?
• Should we encourage evaluators not to update their ratings in response to seeing AI ratings or general characterizations; only to specific identified misunderstandings/omissions.

What evaluators may use

Evaluators may use AI tools selectively. Appropriate uses include looking up or clarifying content in the paper (e.g. using NotebookLM as a reference tool), or running extensive checks that are not feasible to do by hand: in which case the AI-assisted section should appear separately (in a dedicated section, link, or footnote) and the evaluators should include at least a hand-checked sample.

Consider: Where the evaluator lacks expertise in a particular subject, shall we recommend against using AI tools to complement their own knowledge or expertise, except where this involves a simple 'lookup or clarification'? Rationale — we aim to provide a corpus human expertise and judgment, rather than to propagate existing AI model content ('contamination'). While, in principle, experts could augment their understanding by using AI to learn about a specific technique and improve their human advising, this again seems likely to lead to 'contamination'. As a compromise, evaluators could include small ~appendix sections specifying the specific queries they performed, and the insights gined from these, but these should not form the main substance of the report.

Evaluators are encouraged not to use AI tools for overall evaluations or for the ratings and predictions components, except as described above under what we provide.

Transparency requirements

Evaluators must report how AI tools were used by:

1. Providing links to their AI chats or explanations (where possible)
2. Explicitly identifying any sections of text that were generated by LLMs

Human work requirement

Evaluators are expected to put in at least 8 hours of human work, over and above the latent processing time of any LLM tools.

Standing by your evaluation

Evaluators must stand by all content and language in their report as their own judgment. They must independently verify and carefully consider any points raised by AI tools.

Related evidence and context

13 Jul 2026 — this bears updating. One internal update — our own experimentation with late-2025 tools suggested these provided substantial value, and made points that often overlapped with our human evaluators (even though these evaluations should have been out of the model context window).

Previous note; Emerging evidence points toward a complementary role for AI in peer review processes like ours. Biswas et al. (2025), AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot(arXiv:2604.13940), report on a large-scale survey of AAAI-26 authors and program committee members and found "participants not only found AI reviews useful, but actually preferred them to human reviews on key dimensions such as technical accuracy and research suggestions". The system processed all 22,977 full-paper submissions in under a day using frontier models with tool use and safeguards.

This offers promising evidence for AI-assisted peer review at scale. But we are being cautious — we consider we use AI checks as a complement and a supplement to human evaluation, preserving independent expert judgment while adding an additional consistency check and source of suggestions.

Also relevant: Johnny Coates' "Best Practices for preprint peer review services in the use of AI".