Comparing LLM and human reviews of social science research using data from Unjournal.org

Authors

Affiliations

Valentin Klotzbücher

University of Basel & University Hospital Basel

David Reinstein

The Unjournal

Lorenzo Pacchiardi

University of Cambridge, Leverhulme Centre for the Future of Intelligence

Tianmai Michael Zhang

University of Washington

Abstract

We benchmark how frontier LLMs critique, rate, and prioritize research, comparing AI-generated evaluations to human expert assessments from The Unjournal. Human ratings serve as a reference signal—the core goal is measuring and diagnosing AI reasoning quality: understanding calibration, failure modes, and systematic “taste” differences between AI and human evaluators.

Introduction

Include global setup and parameters

source("setup_params.R")

Work in progress

Pages, metrics, and comparisons are under active development. Expect rough edges and frequent updates.

For Grant Reviewers

Looking for our funding proposal? Jump directly to the Proposal chapter for project scope, budget, and deliverables.

Is AI good at peer-reviewing? Does it offer useful and valid feedback? Can it predict how human experts will rate research across a range of categories? How can it help academics do this “thankless” task better? Is it particularly good at spotting errors? Are there specific categories, e.g. spotting math errors or judging real-world relevance, where it does surprisingly well or poorly? How does its “research taste” compare to humans?

If AI research-evaluation works it could free up a lot of scientific resources – perhaps $1.5 billion/year in the US alone (Aczel, Szaszi, and Holcombe 2021) – and offer more continual and detailed review, helping improve research and address the “peer-review crisis”. Numerous accounts report a system under strain, with widespread difficulty finding human peer reviewers with appropriate expertise willing to put in careful effort. An influx of AI-generated or AI-human hybrid research seems likely to exacerbate this problem.¹ AI research evaluation could also help characterize methodological strengths and weaknesses across papers, aiding research training and research direction-setting.

Furthermore, “the leaders of the biggest A.I. labs argue that artificial intelligence will usher in a new era of scientific discovery, which will help us cure diseases and accelerate our ability to address the climate crisis” – e.g., Sam Altman on cures for cancer, etc. Understanding how AI engages with research evaluations may provide a window into its values, abilities, and limitations.²

In this project, we pilot the use of The Unjournal’s evaluations as a benchmark to test the capabilities of current large language models (LLMs) and bespoke agent-based tools, considering how the research evaluations these generate compare to expert human reviews. The Unjournal systematically prioritizes ‘impactful’ research and pays for high-quality human evaluations, structured quantified ratings, claim identification and assessment, and predictions. They further labeled and classified “major consensus critiques” from a subset of these evaluations.³

We next prompt several LLMs and agent-based tools⁴ to review the same set of quantitative social science research evaluated by The Unjournal’s human evaluators, with the same set of criteria and guidelines. Each paper is assessed on specific dimensions – for example, the strength of its evidence, rigor of methods, clarity of communication, openness/reproducibility, relevance to global priorities, and overall quality. The LLM will provide quantitative scores (with uncertainty intervals) on these criteria and produce a written evaluation.

Our initial dataset will include the 60 research papers that have existing Unjournal human evaluations.⁵ For each paper, the AI will generate: (1) numeric ratings on the defined criteria, (2) identification of the paper’s key claims, and (3) a detailed review enumerating and discussing the paper’s contributions and weaknesses. We then compare the AI-generated evaluations to the published human evaluations, considering both quantitative metrics (e.g., inter-rater reliability), identification of consensus critiques, and qualitative comparison. The next phase will train and tune models, and measure their performance on research currently in The Unjournal’s pipeline, i.e., where no human evaluation has yet been made public, to enable out-of-time validation, confirmatory hypothesis tests, and less risk of model contamination.

This complements other recent work on AI-evaluation approaches and benchmarks, outlined below. Our work offers in-depth comparisons and oversight in a high-nuance and high-value context: global-priorities-relevant economics and social-science research, and leverages The Unjournal’s ongoing pipeline as a continual experimental and feedback environment.

Research Goals and Questions (overview)

1. Measure the value of frontier AI research evaluation vs. human peer review
   • Is state-of-the-art AI research evaluation reliable and useful enough to replace human peer review in these contexts?
   • Reverse question: The Unjournal was asked – “Are human evaluations worth it”?
   • How well can AI predict what humans will report?
   • What are AI’s strengths & weaknesses relative to humans, e.g., in terms of: (1) Predicting journal outcomes, citations, etc. (2) Generating feedback deemed most useful by humans (3) Spotting research errors (4) Diagnosing different types of strengths and limitations (e.g., reasoning transparency, justification of methods and assumptions, communication) (5) Judging content without biases/statistical discrimination based on prestige labels, etc.
2. Compare methods for AI research-evaluation, find the best approaches
   • Which approaches come closest to (the desirable features of) human evaluations and in which ways?
   • What gets closest to the ‘frontier’, doing better in terms of outcomes like those mentioned above?
3. The nature of frontier AI ‘preferences & tastes’ over research
   • Which types of papers and approaches do default models tend to ‘like’ more/less relative to humans when given the same guidelines?
   • Do we see signs of characteristic preferences that help us understand how AIs are likely to prioritize, conduct, and evaluate research?
4. Hybrid human-AI performance (future trials)

Our work in context

AI practical research applications⁶

Luo et al. (2025) survey LLM roles from idea generation to peer review, including experiment planning and automated scientific writing. They highlight opportunities (productivity, coverage of long documents) alongside governance needs (provenance, detection of LLM-generated content, standardizing tooling) and call for reliable evaluation frameworks.

Korinek (2025) considers the specific potential for using AI in economic research, offering “working examples and step-by-step code” to help economists use agents for literature reviews, econometric code, fetching and analyzing economic data, and coordinate complex research workflows. He advocates economists using AI tools to boost productivity and “to understand [these tools’] capabilities and limitations.” He calls for human researcher oversight “to ensure [these tools] pursue the right objectives and maintain research integrity”. He outlines a range of current limitations and open questions (outlined here), including hallucinations, mistakes compounding through multi-agent workflows and “brittleness to small variations in prompts”. At the higher level, he argues “the questions that will matter most—what problems deserve investigation, how should we evaluate trade-offs, what constitutes progress—will remain irreducibly human.”

Mitchener, Yiu, et al. (2025) present Kosmos, “an AI scientist for autonomous discovery”. However, evaluations reveal important limitations:

• “Kosmos was found to be only 57% accurate in statements that required interpretation of results, likely due to its propensity to conflate statistically significant results with scientifically valuable ones”

• “although 85% of statements derived from data analyses were accurate, our evaluations do not capture if the analyses Kosmos chose to execute were the ones most likely to yield novel or interesting scientific insights.”… “identifying valuable discoveries Kosmos made is a time-intensive process that relies on human scientists with significant domain expertise”

AI peer review, error detection, and research evaluation

Son et al. (2025) evaluates LLMs on 83 papers paired with 91 confirmed significant scientific errors. This resembles our own approach, although they focus on erratta/retraction-worthy errors, while we consider more nuanced methodological limitations. Their strongest (OpenAI o3) model identified only 21.1% of these errors, with only 6.1% precision.⁷

Eger et al. (2025) provide a broad review of LLMs in science and a focused discussion of AI‑assisted peer review. They argue: (i) peer‑review data is scarce and concentrated in CS/OpenReview venues; (ii) targeted assistance that preserves human autonomy is preferable to end‑to‑end reviewing; and (iii) ethics and governance (bias, provenance, detection of AI‑generated text) are first‑class constraints.

Dycke, Kuznetsov, and Gurevych (2023) present NLPEER, a “multidomain corpus” of more than 5000 papers and 11k review reports from 2012 to 2022 from five different venues”.

Zhang and Abernethy (2025) propose deploying LLMs as quality checkers to surface critical problems instead of generating full narrative reviews. Using papers from WITHDRARXIV and an automatic evaluation framework that leverages “LLM-as-judge,” they find the best performance from top reasoning models but still recommend human oversight.

Pataranutaporn et al. (2025) asked four nearly state-of-the-art LLM models (GPT-4o mini, Claude 3.5 Haiku, Gemma 3 27B, and LLaMA 3.3 70B) to consider 1220 unique papers “drawn from 110 economics journals excluded from the training data of current LLMs”. They prompted the models to act “in your capacity as a reviewer for [a top-5 economics journal]” and make a publication recommendation using a 6-point scale ranging from “1 = Definite Reject…” to “6. Accept As Is…”. They asked it to evaluate each paper on a 10-point scale for originality, rigor, scope, impact, and whether it was ‘written by AI’. They also (separately) had LLMs rate 330 papers with the authors’ identities removed, or replacing the names with fake male/female names and real elite or non-elite institutions or with prominent male or female economists attached.

They compare the LLMs’ ratings with the RePEC rankings for the journals the papers were published in, finding general alignment. They find mixed results on detecting AI-generated papers. In the names/institutions comparisons, they also find the LLMs show biases towards named high-prestige male authors relative to high-prestige female authors, as well as biases towards elite institutions and US/UK universities.

There have been several other empirical benchmarking projects, including work covered in LLM4SR: A Survey on Large Language Models for Scientific Research and Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation. (See cited surveys for additional benchmarks.)

Zhang et al. (2025) use AI conference paper data and LLM agents for pairwise manuscript comparisons, finding this approach outperforms traditional rating-based methods for identifying high-impact papers by citation metrics, though with some evidence of biases favoring established research institutions.⁸

We discuss how our approach differs from prior work below.⁹

AI research taste and potential to guide research

A key motivation for this work stems from broader claims about AI’s potential to accelerate scientific discovery and “revolutionize science.” If AI systems are to contribute meaningfully to research—not just as tools but as evaluators and potential collaborators—we need to understand their “taste”: what they value, what they miss, and whether their priorities align with human expert judgment and real-world research impact.

Understanding LLM preferences and blind spots when evaluating research provides a window into:

• Whether AI can identify truly impactful research vs. stylistic or superficial features
• Potential systematic biases in how AI might direct future research priorities
• The risks and opportunities of AI-assisted or AI-directed research agendas

This connects to broader debates about the returns to science in the presence of technological risk (Clancy 2023), where optimistic forecasts of AI-accelerated science must be weighed against potential downsides and the question of whether AI judgments align with what actually advances knowledge and improves outcomes.

Our present focus, The Unjournal opportunity

Our project distinguishes itself through several unique advantages:

Actual human evaluations: We use actual human evaluations of research in economics and adjacent fields, past and prospective, including both reports, ratings, and predictions.¹⁰ Other work has relied on collections of research and grant reviews, including NLPEER, SubstanReview, and the Swiss National Science Foundation. That data has a heavy focus on computer-science adjacent fields, and is less representative of mainstream research peer review practices in older, established academic fields. Note that The Unjournal commissions the evaluation of impactful research, often from high-prestige working paper archives like NBER, and makes all evaluations public, even if they are highly critical of the paper.

Rich, structured data: The Unjournal’s 55+ evaluation packages enable us to train and benchmark the models. Their detailed written reports and multi-dimensional percentile ratings with credible intervals allow us to compare the ‘taste’, priorities, and comparative ratings of humans relative to AI models across the different criteria and domains.

Ground truth outcomes: The ‘journal tier prediction’ outcomes provide an external ground-truth¹¹ enabling a human-vs-LLM horse race.

Prospective pipeline: The Unjournal’s pipeline of future evaluations allows for clean out-of-training-data predictions and evaluation, serving as a “live testing lab” for a new judging panel, enabling out-of-time testing and validation from authors and human evaluators.

Hybrid evaluation potential: We are also planning multi-armed trials on these human evaluations (cf. Brodeur et al. 2025) to understand the potential for hybrid human-AI evaluation in this context.

The economics, social science, and global-impact policy context provides a unique high-value and nuanced setting to measure and understand AI tendencies and capabilities. According to Korinek (2025) “economists have a special responsibility. We understand incentives, market dynamics, and resource allocation. We grasp the subtleties of human behavior and social coordination. These insights will be crucial for the transition to an AI-augmented research”.

Next steps and future directions

This project is ongoing. Our planned next steps include:

1. Further statistical analyses: Intuitive information theoretic and Bayesian measures and tests; multi-level modeling (random and mixed effects for raters, paper, rating category, etc.)

2. Content-swap ‘bias tests’: Following (Pataranutaporn et al. 2025), test whether LLMs show systematic biases based on author names, institutional affiliations, or other non-content features

3. Predict journal-outcome/bibliometrics horse-race: Compare human vs. LLM predictions against actual publication outcomes

4. Refine LLM model, compare dimensions, test: Systematic comparison of prompts, models, and approaches

5. Explore/analyze descriptive evaluation content: Qualitative analysis of LLM-generated evaluations

6. Human enumerators: Employ human raters to assess whether LLMs identify consensus critiques

7. Train ‘predict human evaluations’ model, preregister for future evaluations: Out-of-time validation with clean test set

8. Possible hybrid human-AI trials: Test whether human-AI collaboration improves on either alone

9. Policy measures and decision criteria: How do we decide and agree upon whether AI, Human, or Hybrid are ‘better’ at this?

10. Implications for The Unjournal/evaluation processes: What can we learn to improve evaluation systems?

Project planning, funding, status

This benchmarking project is a collaboration with The Unjournal, a grant-funded nonprofit¹² that has published 55+ detailed evaluation packages to build open, quantitative evaluation infrastructure and processes, focusing on global-priorities-relevant research in economics, policy, and social science.¹³

Team

Core team:

• David Reinstein (PI): Economist, founder/director of The Unjournal. 25+ years research experience (UC Berkeley, Essex, Exeter, Rethink Priorities). Track record of building evaluation infrastructure and working with impact-focused organizations.
• Valentin Klotzbücher (Co-lead): Economist and statistician (University of Basel). Applied econometrics, causal inference, and reproducible pipelines (R/Quarto). Co-leads current LLM–Unjournal benchmarking work.
• Tianmai Michael Zhang (Technical collaborator): PhD student, University of Washington. Co-author of “Reviewing Scientific Papers for Critical Problems With Reasoning LLMs.” Active contributor to the Black Spatula Project.
• Lorenzo Pacchiardi (Advisor): Research Associate, Leverhulme Centre for the Future of Intelligence, Cambridge. Leads Open Philanthropy–funded project on benchmarking LLM data science tasks. Unjournal advisory board.

We plan to leverage additional support from Unjournal’s management team, operations staff, and evaluator pool (200+ credentialed researchers).

Funding and budget

We are actively seeking funding, in conjunction with The Unjournal, to expand this work beyond the pilot scale to a more robust and rigorous scientific benchmark. The interactive budget model below is linked to this Google Sheet.

Show code

params = ({
  duration_months: [12, 18, 24],
  reinstein_fte: [0.3, 0.5, 0.7],
  reinstein_annual_salary: [120000, 120000, 120000],
  klotz_fte: [0.15, 0.2, 0.3],
  klotz_annual_salary: [120000, 120000, 120000],
  num_eval_packages: [30, 50, 80],
  cost_per_eval_package: [1750, 1750, 1750],
  enumerator_hourly_rate: [50, 50, 50],
  senior_hourly_rate: [100, 100, 100],
  num_papers_consensus: [30, 40, 60],
  hours_per_paper_consensus: [6, 6, 6],
  num_focal_papers_llm: [30, 40, 60],
  num_models_per_paper: [3, 3, 3],
  num_enumerators: [2, 2, 2],
  hours_per_eval_unit: [1.33, 1.33, 1.33],
  fraction_evals_senior_check: [0.33, 0.33, 0.33],
  author_response_incentives: [8000, 10000, 15000],
  crowd_incentives: [10000, 15000, 20000],
  travel_support: [5000, 10000, 15000],
  computing_costs: [15000, 20000, 30000],
  additional_engineering: [20000, 30000, 40000],
  software_server_costs: [5000, 5000, 7000]
})

// Scenario labels
scenarioLabels = ["Small", "Medium", "Large"]

Show code

viewof scenario = Inputs.radio(scenarioLabels, {value: "Medium", label: "Budget scenario"})

Show code

idx = scenarioLabels.indexOf(scenario)

// Helper to get param value for current scenario
p = (name) => params[name][idx]

// Computed costs (matching Google Sheet formulas)
costs = ({
  // People
  reinstein_cost: p("duration_months") * p("reinstein_fte") * p("reinstein_annual_salary") / 12,
  klotz_cost: p("duration_months") * p("klotz_fte") * p("klotz_annual_salary") / 12,

  // Evaluations
  eval_packages: p("num_eval_packages") * p("cost_per_eval_package"),

  // Enumerators
  consensus_critiques: p("num_papers_consensus") * p("hours_per_paper_consensus") * p("enumerator_hourly_rate"),
  llm_eval_enumerators: p("num_focal_papers_llm") * p("num_models_per_paper") * p("num_enumerators") * p("hours_per_eval_unit") * p("enumerator_hourly_rate"),
  senior_checks: p("num_focal_papers_llm") * p("num_models_per_paper") * p("fraction_evals_senior_check") * p("senior_hourly_rate"),

  // Incentives
  author_incentives: p("author_response_incentives"),
  crowd_incentives: p("crowd_incentives"),

  // Other
  travel: p("travel_support"),
  computing: p("computing_costs"),
  engineering: p("additional_engineering"),
  software: p("software_server_costs")
})

// Subtotals
subtotals = ({
  people: costs.reinstein_cost + costs.klotz_cost,
  evaluations: costs.eval_packages,
  enumerators: costs.consensus_critiques + costs.llm_eval_enumerators + costs.senior_checks,
  incentives: costs.author_incentives + costs.crowd_incentives,
  other: costs.travel + costs.computing + costs.engineering + costs.software
})

grandTotal = subtotals.people + subtotals.evaluations + subtotals.enumerators + subtotals.incentives + subtotals.other

// Currency formatter
fmt = d3.format("$,.0f")

Show code

html`<div style="font-size: 0.95em;">
<p><strong>Scenario: ${scenario}</strong> - Duration: ${p("duration_months")} months, ${p("num_eval_packages")} evaluation packages, ${p("num_focal_papers_llm")} focal papers for LLM comparison</p>

<table style="width: 100%; border-collapse: collapse; margin: 1em 0;">
<thead>
<tr style="background: #f0f0f0; border-bottom: 2px solid #333;">
  <th style="text-align: left; padding: 8px;">Section</th>
  <th style="text-align: left; padding: 8px;">Item</th>
  <th style="text-align: right; padding: 8px;">Cost</th>
  <th style="text-align: right; padding: 8px;">Subtotal</th>
</tr>
</thead>
<tbody>
<tr><td style="padding: 6px;"><strong>People</strong></td><td>Research team (co-PIs) @ ${((p("reinstein_fte")+p("klotz_fte"))*100).toFixed(0)}% FTE combined</td><td style="text-align: right;">${fmt(subtotals.people)}</td><td style="text-align: right; font-weight: bold;">${fmt(subtotals.people)}</td></tr>

<tr style="background: #fafafa;"><td style="padding: 6px;"><strong>Evaluations</strong></td><td>${p("num_eval_packages")} packages @ ${fmt(p("cost_per_eval_package"))}</td><td style="text-align: right;">${fmt(costs.eval_packages)}</td><td style="text-align: right; font-weight: bold;">${fmt(subtotals.evaluations)}</td></tr>

<tr><td style="padding: 6px;"><strong>Enumerators</strong></td><td>Consensus critiques (${p("num_papers_consensus")} papers x ${p("hours_per_paper_consensus")}h)</td><td style="text-align: right;">${fmt(costs.consensus_critiques)}</td><td></td></tr>
<tr><td></td><td>LLM eval labeling (${p("num_focal_papers_llm")} papers x ${p("num_models_per_paper")} models x ${p("num_enumerators")} raters)</td><td style="text-align: right;">${fmt(costs.llm_eval_enumerators)}</td><td></td></tr>
<tr><td></td><td>Senior expert checks (${(p("fraction_evals_senior_check")*100).toFixed(0)}% sample)</td><td style="text-align: right;">${fmt(costs.senior_checks)}</td><td style="text-align: right; font-weight: bold;">${fmt(subtotals.enumerators)}</td></tr>

<tr style="background: #fafafa;"><td style="padding: 6px;"><strong>Incentives</strong></td><td>Author response incentives</td><td style="text-align: right;">${fmt(costs.author_incentives)}</td><td></td></tr>
<tr style="background: #fafafa;"><td></td><td>Crowd/prediction market incentives</td><td style="text-align: right;">${fmt(costs.crowd_incentives)}</td><td style="text-align: right; font-weight: bold;">${fmt(subtotals.incentives)}</td></tr>

<tr><td style="padding: 6px;"><strong>Other</strong></td><td>Travel & conferences</td><td style="text-align: right;">${fmt(costs.travel)}</td><td></td></tr>
<tr><td></td><td>LLM/compute costs</td><td style="text-align: right;">${fmt(costs.computing)}</td><td></td></tr>
<tr><td></td><td>Additional engineering</td><td style="text-align: right;">${fmt(costs.engineering)}</td><td></td></tr>
<tr><td></td><td>Software/server overhead</td><td style="text-align: right;">${fmt(costs.software)}</td><td style="text-align: right; font-weight: bold;">${fmt(subtotals.other)}</td></tr>

<tr style="border-top: 2px solid #333; background: #e8f4e8;">
  <td style="padding: 8px;"><strong>TOTAL</strong></td>
  <td></td>
  <td></td>
  <td style="text-align: right; font-weight: bold; font-size: 1.1em;">${fmt(grandTotal)}</td>
</tr>
</tbody>
</table>
</div>`

Key parameters driving this budget

Show code

html`<table style="width: 100%; font-size: 0.9em;">
<tr><th style="text-align: left;">Parameter</th><th>Small</th><th>Medium</th><th>Large</th><th style="text-align: left;">Description</th></tr>
<tr><td>Duration</td><td>${params.duration_months[0]} mo</td><td>${params.duration_months[1]} mo</td><td>${params.duration_months[2]} mo</td><td>Project duration</td></tr>
<tr><td>Research team FTE</td><td>${(params.reinstein_fte[0]+params.klotz_fte[0]).toFixed(2)}</td><td>${(params.reinstein_fte[1]+params.klotz_fte[1]).toFixed(2)}</td><td>${(params.reinstein_fte[2]+params.klotz_fte[2]).toFixed(2)}</td><td>Combined co-PI effort allocation</td></tr>
<tr><td>Eval packages</td><td>${params.num_eval_packages[0]}</td><td>${params.num_eval_packages[1]}</td><td>${params.num_eval_packages[2]}</td><td>New Unjournal evaluation packages</td></tr>
<tr><td>Focal papers</td><td>${params.num_focal_papers_llm[0]}</td><td>${params.num_focal_papers_llm[1]}</td><td>${params.num_focal_papers_llm[2]}</td><td>Papers for detailed LLM comparison</td></tr>
<tr style="background: #f9f9f9;"><td colspan="5"><em>Enumerator costs</em></td></tr>
<tr><td>Enumerator rate</td><td>$${params.enumerator_hourly_rate[0]}/hr</td><td>$${params.enumerator_hourly_rate[1]}/hr</td><td>$${params.enumerator_hourly_rate[2]}/hr</td><td>Hourly rate for labeling work</td></tr>
<tr><td>Senior expert rate</td><td>$${params.senior_hourly_rate[0]}/hr</td><td>$${params.senior_hourly_rate[1]}/hr</td><td>$${params.senior_hourly_rate[2]}/hr</td><td>Rate for expert review checks</td></tr>
<tr><td>Hours per eval unit</td><td>${params.hours_per_eval_unit[0]}</td><td>${params.hours_per_eval_unit[1]}</td><td>${params.hours_per_eval_unit[2]}</td><td>Time to label each LLM evaluation</td></tr>
</table>`

Key cost categories include: Core team time (PI + co-lead), engineering and development, payments for human research evaluators (extending The Unjournal’s pipeline), payments for enumerators/experts to label and provide human feedback on LLM evaluator claims and critiques, and compute/API costs. No model training or large-compute capabilities work—all spending focuses on evaluation, tooling, and human experimentation.

Deliverables

1. Living benchmark dataset: Continuously updated papers + human evaluations + LLM outputs + adjudication scores, openly available throughout the project
2. Public dashboard: Updatable website where new models can be compared against evolving benchmarks
3. Open-source codebase: Pipelines for running AI evaluations and computing metrics
4. Failure-mode taxonomy: Documented patterns of AI reasoning failures
5. “Taste/alignment” analysis: Quantitative comparison of human vs. AI research preferences
6. Academic paper(s): Targeting venues like Nature Human Behaviour, Science, or field-relevant journals
7. Practical guidance: Playbook for when AI peer review helps and where it fails

Success metrics

Benchmark quality:

• 80+ papers with complete paired human/LLM evaluations
• 3+ model families systematically compared
• Dataset and code publicly released with documentation

Informative results:

• Pre-registered quantitative metrics with adequate statistical power
• At least one publishable paper or equivalent preprint
• Clear evidence of specific failure modes (or their absence)

Practical uptake:

• Unjournal team finds tools useful for triage and evaluation
• 2+ external organizations (GCR funders, journals, research users) engage with outputs
• If tools prove not useful, we report this explicitly as a negative result

Risks and mitigations

Risk	Mitigation
LLMs perform poorly	Negative results are valuable; focus on taxonomy and practical guidance
Human evaluations are noisy	Use adjudication, curated subsets, pre-registration
Engineering bottlenecks	Modular pipelines, flexible staffing, leverage existing tools
Low external uptake	Build relationships with GCR organizations early; design for usability

Why now?

LLMs are only now sophisticated enough to plausibly conduct research evaluation–but the window to understand their failure modes before widespread adoption is closing. The Unjournal’s structured evaluations provide a unique “ground truth” for benchmarking AI reasoning in economics and policy research; no comparable dataset exists. If AI research evaluation becomes reliable, it could reduce costs and accelerate evidence synthesis for time-sensitive decisions. If it remains unreliable, decision-makers need to know.

Project timeline

The project follows an iterative, overlapping structure rather than sequential phases. Human evaluations flow continuously from The Unjournal’s pipeline, analysis proceeds alongside data collection, and outputs are shared as living documents throughout.

Show code

timelineData = [
  {activity: "Pre-analysis plan & registration", start: 1, end: 3, row: 0, color: "#6b7280"},
  {activity: "Pipeline & tooling development", start: 1, end: 6, row: 1, color: "#3b82f6"},
  {activity: "Baseline LLM runs (existing evals)", start: 2, end: 5, row: 2, color: "#8b5cf6"},
  {activity: "Human evaluations (ongoing intake)", start: 1, end: idx === 0 ? 12 : idx === 1 ? 18 : 24, row: 3, color: "#10b981"},
  {activity: "LLM evaluation of new papers", start: 4, end: idx === 0 ? 10 : idx === 1 ? 15 : 21, row: 4, color: "#8b5cf6"},
  {activity: "Enumerator labeling & adjudication", start: 5, end: idx === 0 ? 11 : idx === 1 ? 16 : 22, row: 5, color: "#f59e0b"},
  {activity: "Analysis & model refinement", start: 3, end: idx === 0 ? 12 : idx === 1 ? 18 : 24, row: 6, color: "#ef4444"},
  {activity: "Training predictive models", start: idx === 0 ? 6 : idx === 1 ? 8 : 10, end: idx === 0 ? 10 : idx === 1 ? 15 : 20, row: 7, color: "#ec4899", show: idx >= 1},
  {activity: "Out-of-time validation (pipeline)", start: idx === 0 ? 9 : idx === 1 ? 13 : 17, end: idx === 0 ? 12 : idx === 1 ? 18 : 24, row: 8, color: "#06b6d4", show: idx >= 1},
  {activity: "Dashboard & data releases", start: 4, end: idx === 0 ? 12 : idx === 1 ? 18 : 24, row: 9, color: "#84cc16"},
  {activity: "Publications & dissemination", start: idx === 0 ? 8 : idx === 1 ? 12 : 16, end: idx === 0 ? 12 : idx === 1 ? 18 : 24, row: 10, color: "#6b7280"}
]

maxMonth = idx === 0 ? 12 : idx === 1 ? 18 : 24
filteredTimeline = timelineData.filter(d => d.show !== false)

Show code

html`<div style="font-size: 0.9em; overflow-x: auto;">
<p><strong>${scenario} scenario timeline (${maxMonth} months)</strong></p>
<svg width="${Math.max(700, maxMonth * 35 + 250)}" height="${filteredTimeline.length * 28 + 40}">
  <!-- Month labels -->
  ${Array.from({length: maxMonth}, (_, i) => i + 1).map(m =>
    `<text x="${250 + (m - 0.5) * 35}" y="15" text-anchor="middle" font-size="11" fill="#666">${m}</text>`
  ).join('')}

  <!-- Grid lines -->
  ${Array.from({length: maxMonth + 1}, (_, i) => i).map(m =>
    `<line x1="${250 + m * 35}" y1="20" x2="${250 + m * 35}" y2="${filteredTimeline.length * 28 + 30}" stroke="#e5e7eb" stroke-width="1"/>`
  ).join('')}

  <!-- Activity bars -->
  ${filteredTimeline.map((d, i) => `
    <g transform="translate(0, ${25 + i * 28})">
      <text x="245" y="14" text-anchor="end" font-size="11" fill="#333">${d.activity}</text>
      <rect x="${250 + (d.start - 1) * 35}" y="2" width="${(d.end - d.start + 1) * 35 - 4}" height="18" rx="3" fill="${d.color}" opacity="0.8"/>
    </g>
  `).join('')}
</svg>
</div>`

Key timeline principles:

• Pre-registration early: We specify and register a general pre-analysis plan in months 1–3 to constrain researcher degrees of freedom, with amendments documented as the project evolves.
• Continuous human evaluations: New Unjournal evaluation packages arrive throughout the project, feeding both the benchmark dataset and (in larger scenarios) model training.
• Analysis throughout: Statistical analysis and model refinement proceed iteratively as data accumulates—we don’t wait for “all data” before analyzing.
• Living research: Data and results are openly available and updated continuously; there is no “final dataset,” only versioned releases with clear provenance.

Show code

predictiveNote = idx >= 1 ? `<p style="margin-top: 0.5em;"><strong>Predictive modeling</strong> (Medium/Large): In months ${idx === 1 ? '8-15' : '10-20'}, we train models on accumulated human evaluations to predict ratings for papers entering The Unjournal pipeline, enabling out-of-time validation in the final phase.</p>` : ''
html`${predictiveNote}`

For more details, see The Unjournal and our project repository.¹⁴

Aczel, Balazs, Barnabas Szaszi, and Alex O Holcombe, “A billion-dollar donation: Estimating the cost of researchers’ time spent on peer review,” Research integrity and peer review, 6 (2021), 1–8 (Springer).

Brodeur, Abel, David Valenta, Alexandru Marcoci, Juan P Aparicio, Derek Mikola, Bruno Barbarioli, Rohan Alexander, Lachlan Deer, Tom Stafford, Lars Vilhuber, and others, “Comparing human-only, AI-assisted, and AI-led teams on assessing research reproducibility in quantitative social science,” Institute for Replication (I4R) Discussion Paper, 195 (2025).

Clancy, Matt, “The returns to science in the presence of technological risk,” arXiv preprint arXiv:2312.14289, (2023).

Dycke, Nils, Ilia Kuznetsov, and Iryna Gurevych, “NLPeer: A unified resource for the computational study of peer review,” in Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, eds., 2023.

Eger, Steffen, Yong Cao, Jennifer D’Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, Chenghua Lin, Nafise Sadat Moosavi, Wei Zhao, and Tristan Miller, “Transforming science with large language models: A survey on AI-assisted scientific discovery, experimentation, content generation, and evaluation,” arXiv preprint arXiv:2505.05151, (2025).

Korinek, Anton, “AI agents for economic research,” SSRN Working Paper, (2025).

Luo, Ziming, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du, “LLM4SR: A survey on large language models for scientific research,” arXiv preprint arXiv:2501.04306, (2025).

Mitchener, Ludovico, Ethan Yiu, and others, “Kosmos: An AI scientist for autonomous discovery,” arXiv preprint arXiv:2511.02824, (2025).

Pataranutaporn, Pat, Nattavudh Powdthavee, Chayapatr Achiwaranguprok, and Pattie Maes, “Can AI solve the peer review crisis? A large scale cross model experiment of LLMs’ performance and biases in evaluating over 1000 economics papers,” 2025.

Son, Jaeho, and others, “FLAWS: A benchmark for error identification and localization in scientific papers,” arXiv preprint arXiv:2511.21843, (2025).

Zhang, Tianmai M, and Neil F Abernethy, “Reviewing scientific papers for critical problems with reasoning LLMs: Baseline approaches and automatic evaluation,” arXiv preprint arXiv:2505.23824, (2025).

Zhang, Yaohui, Haijing Zhang, Wenlong Ji, Tianyu Hua, Nick Haber, Hancheng Cao, and Weixin Liang, “From replication to redesign: Exploring pairwise comparisons for LLM-based peer review,” arXiv preprint arXiv:2506.11343, (2025).

---

1. E.g., see anectodal reports. Proposed solutions include a credit or payment-based model. (Further references to be added.).

2. Eger et al. (2025): “When it comes to using AI tools for science, ethics is of overarching importance. This is because the AI tools exhibit various limitations, e.g., they (i) may hallucinate and fabricate content, (ii) exhibit bias, (iii) may have limited reasoning abilities and (iv) sometimes lack suitable evaluation, … among many other concerns such as risks of fake science, plagiarism, and lack of human authority. Indeed, the European Union has recently released guidelines on the responsible use of AI for science. In it, it points out that, while”[r]esearch is one of the sectors that could be most significantly disrupted by generative AI” where “AI has great potential for accelerating scientific discovery and improving the effectiveness and pace of research and verification processes”

3. An individual rater (David Reinstein, Co-director of The Unjournal) curated a pilot version of this here, focusing on critiques identified by multiple evaluators and/or confirmed by the author or evaluation manager. We are currently (Jan 5, 2026) comparing this enumeration to the issues identified by LLM’s (see “Results: Critiques” chapter). Future work will systematically employ multiple enumerators with a pre-defined rubric.

4. Dec 2025 focus: OpenAI’s GPT-5.2 Pro model

5. This is slightly off as of January 2026. We only have 56 papers with official Unjournal evaluations. I suspect this data includes some papers that are either duplicated or in our pipeline for given an independent non-official evaluation

6. We maintain a NotebookLM of the most relevant and related research here.

7. They asked domain experts to do qualitative analysis, which found most false positives were hallucinations or student-level misconceptions

8. Key findings from Zhang et al. (2025): (1) Uses AI conference paper data; (2) Employs LLM agents for pairwise manuscript comparisons; (3) Significantly outperforms traditional rating-based methods in identifying high-impact papers by citation metrics; (4) Shows some evidence of biases/statistical discrimination based on characteristics like ‘papers from established research institutions’.

9. See below. Claude’s brief summary “Our approach differs from prior work by (i) focusing on structured, percentile-based quantitative ratings with credible intervals that map to decision-relevant dimensions used by The Unjournal; (ii) comparing those ratings to published human evaluations rather than using LLM-as-judge; and (iii) curating contamination-aware inputs (paper text extraction with reference-section removal and token caps), with a roadmap to add multi-modal checks when we score figure- or table-dependent criteria”.

10. For additional context on The Unjournal’s approach and evaluation criteria, see our evaluator guidelines.

11. These represent ground truths about verifiable publication outcomes, not about the ‘true quality’ of the paper.

12. Funders include the Survival and Flourishing Fund, the Long Term Future Fund, and EA Funds; they make all grant information public here.

13. This section was drafted with AI assistance and reviewed by the authors.

14. Methodological implications we plan to adopt from the literature: - Report stability (repeat runs / seeds) and calibration diagnostics alongside point scores. - Keep error verification distinct from quality scoring; when we do verification, prefer author- or expert-validated ground truth over LLM-as-judge, or clearly disclose the judge and its limits. - Track input format effects (PDF vs LaTeX vs extracted text) and costs/latency, since these affect reproducibility and deployment.