Include global setup and parameters
source("setup_params.R")Just Ask the Model: One-Shot LLM Research Evaluation and Structured Expert Review
A working paper comparing structured one-shot research evaluations by GPT-5 Pro with paid expert review packages from The Unjournal, on percentile ratings and narrative critiques of economics and social-science working papers.
Abstract
Peer review is strained, and AI tools generating referee-like feedback are already adopted by researchers and commercial services—yet field evidence on how reliably frontier LLMs can evaluate research remains scarce. We compare structured one-shot evaluations by GPT-5 Pro against paid expert review packages from The Unjournal, an open evaluation platform covering economics and social-science working papers, where both humans and the model rate papers on seven percentile criteria with uncertainty intervals and provide narrative critiques. Treating human evaluations as a high-quality but noisy reference signal, we find that GPT-5 Pro approaches the agreement levels observed among human evaluators themselves on several criteria, while exhibiting consistent failure modes: compressed rating scales, uneven criterion coverage, and variable identification of expert-flagged concerns. A minimal one-shot setup—a single prompt with a fixed rubric, no iteration or retrieval augmentation—yields ratings associated with the human panel, supporting further tests of supplementary use. Uncertainty in the human benchmark and assumptions underlying the reliability adjustment prevent an equivalence claim. Appendix results describe five additional models with differing sample coverage.
Introduction
NoteWhat the date above means
The date is the last time any page of this paper changed. It is not the date of the main analysis.
- July 2025 to March 2026: main analysis and writing by the authors. The GPT-5 Pro and GPT-5.2 Pro runs used in the main results were done in January 2026.
- July 2026: exploratory headless Codex pilot; signposting between the project sites.
- September 2026: corrections to numerical checks and comparisons.
- October 2026: AI-generated related-work note (OpenAI Codex), labelled where it appears.
- November 2026: provisional, AI-scored persistent-miss diagnostic (Claude Opus 5.5), added as an appendix and not yet reviewed by the authors.
Since September 2026, most changes have been made by AI agents at David Reinstein’s request. AI-written material is labelled where it appears.
NoteChoose the right project view
- Current working paper: this is the authoritative version of the statistical analysis and academic presentation. It is available at GitHub Pages and the primary Netlify URL.
- Full project workspace and grant proposal: see the broader project site for project goals, team and funding context, earlier analysis, and appendices. Grant reviewers can go directly to the proposal. This material is retained for context and is not the authoritative source for the latest results.
- LLM–human critique concordance: see the public concordance evidence site for the exploratory issue-mapping judgments and methods notes.
We are also maintaining a small headless Codex evaluation pilot as an exploratory extension. It is deliberately separated from the main API-based evidence because subscription-backed headless runs are useful for low-cost exploration but are less standardized and less directly replicable than the native API workflow.
A collaboration with The Unjournal (55+ open evaluation packages for global-priorities research). Funding: Survival and Flourishing Fund, Long Term Future Fund, EA Funds.
TipCan the model find the issues experts find?
In 12 eligible paper comparisons, GPT-5.2 Pro identified 80% of the curated human consensus issues on average (range: 50–100%). Conversely, 52% of the issues raised by the model matched the curated human concern set. The comparison is exploratory and currently uses GPT-5.2 Pro as the judge; unmatched model concerns are not necessarily wrong.
NoteRelated work: King and Worthington (2026) · AI-generated note, October 23, 2026
King and Worthington (2026) report strong AI review performance against paid rapidPeer experts. This substantially overlaps with our high-quality human comparison; our clearest distinctions concern structured research ratings, uncertainty, and proposed prospective and human–AI studies. See the AI-generated comparison and research priorities and discussion.
Peer review is under strain. Reviewers are hard to find, turnaround times are lengthening, and the system costs an estimated $1.5 billion per year in the United States alone (Aczel, Szaszi, and Holcombe 2021). At the same time, generative AI lowers the cost of producing polished manuscripts; in at least some fields, editors report submission growth that exceeds reviewer capacity, and explicitly link this trend to LLM-assisted writing (Spitzer 2026). This combination creates demand for automated support in editorial and pre-submission workflows.
Commercial AI reviewer products such as Refine (Refine n.d.) and QED Science (QED Science 2026) already market automated referee-like feedback directly to authors, while disclaiming that they substitute for human peer review. Their internal architectures are not publicly documented. Other tools are open. IsItCredible, run by The Catalogue of Errors Ltd, sold automated “Reviewer 2” reports before retiring its hosted service, and has released that pipeline and its prompts as open-source code (The Catalogue of Errors Ltd 2026; IsItCredible.com n.d.). OpenAIReview (Hsu and Tan 2026) is an open-source pipeline whose “progressive” method reads a paper in sequence while keeping a running summary of its definitions, equations, and key claims, at roughly $4 per paper. Meanwhile, publishers are formalizing policies that restrict reviewer use of general-purpose AI tools1 while permitting controlled in-house applications for screening tasks (Leung 2026; Elsevier 2026). Our study complements these efforts with a baseline: how closely the simplest possible setup—a single prompt to a frontier model, with no bespoke pipeline—matches structured expert evaluations, both in its ratings and in the issues it raises.
These developments make the evidentiary gap salient: funders, editors, and policymakers need to know when AI evaluation outputs are trustworthy enough to use, and when they are unstable, biased, or manipulable. Recent work documents three interlocking concerns.2 Reproducibility can be “jagged” across models and time (Thomas, Romasanta, and Pujol Priego 2026), and subtle task reframings can induce systematic output shifts reminiscent of specification search (Asher et al. 2026). Adversarial manipulation is not hypothetical: invisible prompt-injection text can inflate LLM review scores in simulated peer review (Choi et al. 2026). And even without manipulation, AI reviews tend to be less thematically diverse and less focused on interpretation and originality than human reviews (Rajakumar et al. 2026), while LLM scoring exhibits range restriction and halo effects that distort agreement metrics (Wang et al. 2025). The most direct recent comparison is more favorable to AI: on 124 rapidPeer submissions, King and Worthington (2026) find that AI reviews win every decided overall comparison against paid, carefully selected human reviews when an AI model judges, and 26 of 30 judgments by three former journal editors (see our comparison with that study).
The central question we address is therefore: how reliably can frontier LLMs evaluate research, relative to expert peer review and under realistic levels of rater disagreement?3 We study this question in a setting designed to make “expert judgment” observable and multi-dimensional rather than implicit.
We use The Unjournal’s structured human evaluations as a reference signal. We prompt GPT-5 Pro—a frontier reasoning model—with the same rubric and guidelines used by human evaluators, then compare its ratings with available expert evaluations. The main run covers 60 economics and social-science working papers, with 47 human-matched papers; critique overlap is a separate GPT-5.2 Pro analysis of 12 eligible pairings. We ask whether a frontier LLM evaluation can approximate expert judgment, where systematic differences arise, and whether the model reveals characteristic AI preferences over research. GPT-5 Pro shows positive rank agreement with the human panel, alongside compressed ratings and uneven coverage of expert concerns. A paired bootstrap places uncertainty around an approximate comparison with human inter-rater agreement4; it does not establish equivalence to an additional expert. Appendix A reports results from five additional LLMs spanning different capability and cost tiers, with sample availability varying by model. Because the model ranks papers broadly like the human panel but compresses its ratings and misses some expert concerns, it may be more useful alongside human evaluators than in place of them. Testing this directly would mean, for example, checking whether adding a model’s ratings to a human panel improves agreement with held-out evaluations or with later publication outcomes.
Our approach is minimal by design: each model receives the same PDF and a fixed rubric in a single prompt, with no iteration, retrieval augmentation, chain-of-thought scaffolding, or multi-step agentic loop. This provides a baseline for testing more elaborate evaluation procedures. Structured measurement tools,5 steps that verify each flagged problem,6 and prompt-sensitivity controls7 could improve performance, but added complexity could also amplify shared errors or introduce new biases. Their incremental value requires comparison on held-out papers.
The Unjournal setting is particularly well suited for this comparison. It commissions paid expert evaluations using a structured rubric covering seven percentile criteria with 90% credible intervals plus journal-tier predictions, and publishes the resulting packages openly rather than making binary accept/reject decisions—which may increase reviewer effort through accountability and transparency. The resulting ratings and critiques still exhibit substantial inter-rater variation;8 accordingly, we treat human evaluations as a high-quality but noisy reference signal, not ground truth. The rich, multi-dimensional data allow us to compare the priorities and calibration of humans and AI models across criteria and domains, while eventual publication outcomes for journal-tier predictions provide an external validation opportunity9 enabling a human-vs-LLM horse race. Finally, The Unjournal’s pipeline of future evaluations allows for clean out-of-training-data predictions, serving as a live testing lab for prospective validation.
NoteUnjournal impact: author engagement evidence
Do authors engage with and respond to Unjournal evaluations? An April 2026 tracking file covers 57 evaluations, but adjustment status had been assessed for 22 rather than the full set. Within that partly assessed snapshot, 16 papers have a recorded formal written response, at least 5 are coded as showing clear evidence of substantive revision, and 7 authors stated an intention to update. A full staff reassessment remains an ongoing goal.
See the full tabulation: did authors adjust their papers? for the interactive table and public author statements. For broader context, see Evidence: do authors engage with evaluations? in The Unjournal’s knowledge base.
For example, Elsevier’s policy states that “reviewers must not upload submitted manuscripts, or any information from them, into AI Tools” (Elsevier 2026, Reviewers FAQ).↩︎
King and Worthington (2026) summarize a wider set: hallucinated claims and fabricated sources (Ye et al. 2024), failure to recognize novelty (Shin et al. 2025), bias toward prestigious authors and institutions (Hosseini and Horbach 2023; Macharla Vasu et al. 2026), excessive leniency (Russo Latona et al. 2024), hidden-prompt manipulation (Lin 2025), and confidentiality breaches (Naddaf 2025). Field evidence is also growing: the AAAI-26 conference added one labeled AI review to each of 22,977 papers sent for full review (Biswas et al. 2026); randomized studies examine LLM feedback to reviewers (Thakkar et al. 2026) and reviewer AI-use policies at ICML 2026 (Kim et al. 2026); and blinded editors at a cardiology journal rated AI reviews above human reviews on five of seven criteria (Zancanaro et al. 2026). We cite these as King and Worthington describe them and have not yet assessed each study ourselves.↩︎
A second, more exploratory question is how LLMs’ research reasoning and priorities compare with those of human experts: which criteria and features of a paper they weight, and whether they would choose the same research to evaluate. The grant proposal describes planned work on this, including comparing AI prioritization with The Unjournal’s internal prioritization decisions.↩︎
We resample papers with replacement 2,000 times. In each draw we recompute both the model–human and the human–human correlation on the same resampled papers, and take their difference after a reliability adjustment. The spread of that difference across draws gives the interval. See Results and Methods.↩︎
Asirvatham, Mokski, and Shleifer (2026) present GABRIEL, a package that uses GPT to rate attributes of text, and report performance generally indistinguishable from human annotators across more than 1,000 tasks. Their applications are not research evaluation; we cite the paper as evidence that carefully structured LLM rating can match human raters, not as a tested way to improve research evaluation.↩︎
Zhang and Abernethy (2025) treat LLMs as manuscript quality checkers that list a paper’s most critical problems, scored against the stated reasons for withdrawn arXiv papers. They test single-pass checking and suggest a propose-then-verify workflow, in which flagged problems are checked again, for future work.↩︎
Asher et al. (2026) run AI coding agents on the same empirical analyses with neutral prompts and with prompts that nudge toward statistically significant results, to see whether the reported estimates shift.↩︎
On the overall rating, two human evaluators of the same paper have a rank correlation of 0.33 (95% CI 0.02 to 0.59; 39 papers) and a mean absolute difference of 11.6 percentile points; see Results. This is in line with journal peer review generally: a meta-analysis of 48 studies found a mean inter-rater reliability of about 0.34 (intraclass correlation or r²) and a mean Cohen’s kappa of 0.17 (Bornmann, Mutz, and Daniel 2010), though these measures are not identical to ours.↩︎
These represent verifiable publication outcomes, not statements about the “true quality” of the paper.↩︎