Appendix E — King and Worthington (2026): implications for this project

AI-generated research note · October 23, 2026

Written by OpenAI Codex at David Reinstein’s request, based on the October 22, 2026 paper and supplementary appendix. This is an AI-generated interpretation and proposed research agenda, not a new empirical result or an agreed revision of the team’s plans.

Assessment. The study by King and Worthington (2026) substantially overlaps with this project’s motivation and weakens novelty claims based on comparing AI with unusually good, paid human reviewers. The existing Unjournal study still contributes a transparent baseline for structured research ratings and critique agreement. The strongest case for further work is to establish when AI judgments are correct, calibrated, and useful for consequential research decisions, and what additional human effort buys.

What the study establishes

King and Worthington analyze 124 rapidPeer submissions from 2019–2026, with 134 commissioned human reviews. They report generating reviews with Claude Opus 4.8 at maximum reasoning effort, using either the original reviewer instructions without tools or a tailored prompt with web retrieval. GPT-5.5 judges the comparisons; these model names and settings are as reported in the paper. Three former journal editors also judge ten randomly selected review pairs. The human reviewers were paid, carefully selected, and asked for comprehensive, constructive feedback (main text, sections 2–3).

The AI reviews win every decided overall comparison in the headline AI-judged analysis and 26 of 30 editor judgments (87%). Those 30 judgments concern ten pairs assessed by three editors, not 30 independent manuscripts. The editors see the reviews without the manuscripts (supplement, Appendix A). Their choices therefore support the perceived quality of the reviews, but do not independently verify their technical claims against the underlying research.

The evidence goes beyond preferences over fluent prose. Checks address review length, rewriting human reviews in AI style, judging with manuscript access, submission year, stochastic variation, and different generating and judging model families. A blinded audit flags 5.5% of AI claim assessments and 15.9% of human claim assessments as contradicted by the manuscript; a human check agrees on 87 of 90 sampled claims. Appendix F.7 and Table S6 define this error measure narrowly: opinions, suggestions, and claims not verifiable from the manuscript remain in the denominator without counting as errors. It is not a measure of the validity or usefulness of every critique. Moreover, 46 of 124 AI reviews contain at least one flagged error. The controlled-error experiment shows detection of explicit inserted contradictions; it does not establish equivalent sensitivity to subtle failures of identification or external validity.

Version detail. The main text reports 132/132 decided overall AI comparisons, whereas Figure 1 labels this 134/134. This note preserves the common qualitative finding and flags the denominator discrepancy rather than silently resolving it. Sources: paper, sections 2–4 and Figure 1; supplement, Appendices A and F.4–F.10.

Where the Unjournal project differs

Ratings of research and their uncertainty. The working paper compares seven percentile ratings, uncertainty intervals, and journal-tier predictions under the same rubric. King and Worthington’s seven headline areas assess the quality of reviews; they are not seven ratings of the underlying papers. A convincing report can coexist with compressed scores, unstable rankings, or poorly calibrated forecasts. Our current agreement estimates describe these patterns; they do not establish true research quality or calibrated uncertainty.

Correct, important critiques and complementary information. Our critique analysis asks which human concerns the model recovers and which model concerns have no human match. This is a different outcome from overall review preference, but it is not yet a decisive advantage: the current analysis has only 12 eligible pairings and uses an LLM judge. An unmatched model critique can be valuable, and a shared critique can be wrong. The useful extension is blinded, independent adjudication of correctness, severity, actionability, and genuinely new information for both human and AI concerns, including checks of data or code where relevant. Their claim audit makes a stronger validation standard necessary for ours.

Prospective predictions and decisions. The proposed pipeline can register predictions before human evaluations or publication outcomes are available. It should also check earlier public versions and reviews; a new intake date alone does not prevent contamination. Publication venues are verifiable outcomes, with their own selection mechanisms and biases, rather than ground truth about scientific merit. Research prioritization adds a separate question: which work deserves scarce evaluation effort because it could inform important decisions? Agreement with internal prioritization is descriptive evidence, not proof of social value.

Human–AI collaboration and downstream effects. King and Worthington explicitly leave publication decisions, improvements in subsequent revisions, and collective effects unmeasured. Proposed Unjournal trials could compare unaided human evaluation, AI-only evaluation, and human evaluation with AI assistance. Relevant outcomes include independently validated issue detection, harmful false positives, evaluator time, author revisions, and usefulness to research users. Author satisfaction and textual revision alone would not establish scientific improvement. These trials and broader impact measures remain proposed work.

The combination of these questions is a credible contribution. Paid expert reports, broad social-science relevance, or a simple no-tools baseline alone are weak differentiators: rapidPeer already provides substantial overlap on those features. The current study’s smaller sample and older model runs also limit claims about today’s best achievable review quality. Its findings and rapidPeer’s can both hold because the tasks, models, and outcome measures differ.

Proposed priorities for further research

Validate a focused set of consequential critiques. Use independent domain experts, blinded to review source, to adjudicate AI-only, human-only, and shared concerns. Preserve disagreements and distinguish factual contradictions from contested methodological judgments. Prioritize serious errors and useful additions over another overall preference score.
Register a prospective comparison. Freeze manuscripts, prompts, model identifiers, access to tools and supplementary material, predictions, and scoring rules. Compare a current fixed-rubric baseline with a stronger retrieval-assisted approach on held-out papers. Cluster uncertainty by paper when multiple reviews or judges share a manuscript; report ties, missing judgments, and sensitivity to judge choice.
Estimate the value of human effort after AI assistance. In a randomized pilot, measure additional validated issues and useful decisions per unit of evaluator time and total cost, including adjudication and correction costs. Specify contamination controls, assignment units, and primary outcomes before collecting results.
Test preferences and resistance to manipulation. Vary prestige cues, prompt framing, and adversarial instructions while holding research content fixed. Measure whether assistance homogenizes judgments or suppresses useful minority objections. These address concerns King and Worthington also raise; they are motivated extensions, not untouched topics.

Funding implication (AI assessment). A smaller, well-validated prospective study would now be easier to justify than a large expansion devoted mainly to repeating human–LLM agreement comparisons. The existing proposal’s structured measurement, adjudication, prospective validation, prioritization, and collaboration components remain useful, but their value depends on executing them. This note does not change the proposal’s budget or claim those studies have been completed.

See the current working paper, discussion, and broader project and grant proposal. The working paper remains the source for current results; the broader site preserves earlier project context.

# King and Worthington (2026): implications for this project {#sec-peerless-context} ::: {.callout-note} ## AI-generated research note · October 23, 2026 Written by OpenAI Codex at David Reinstein's request, based on the October 22, 2026 paper and supplementary appendix. This is an AI-generated interpretation and proposed research agenda, not a new empirical result or an agreed revision of the team's plans. ::: **Assessment.** The study by @kingWorthington2026Peerless substantially overlaps with this project's motivation and weakens novelty claims based on comparing AI with unusually good, paid human reviewers. The existing Unjournal study still contributes a transparent baseline for structured research ratings and critique agreement. The strongest case for further work is to establish when AI judgments are correct, calibrated, and useful for consequential research decisions, and what additional human effort buys. ## What the study establishes King and Worthington analyze 124 rapidPeer submissions from 2019–2026, with 134 commissioned human reviews. They report generating reviews with Claude Opus 4.8 at maximum reasoning effort, using either the original reviewer instructions without tools or a tailored prompt with web retrieval. GPT-5.5 judges the comparisons; these model names and settings are as reported in the paper. Three former journal editors also judge ten randomly selected review pairs. The human reviewers were paid, carefully selected, and asked for comprehensive, constructive feedback (main text, sections 2–3). The AI reviews win every decided overall comparison in the headline AI-judged analysis and 26 of 30 editor judgments (87%). Those 30 judgments concern ten pairs assessed by three editors, not 30 independent manuscripts. The editors see the reviews without the manuscripts (supplement, Appendix A). Their choices therefore support the perceived quality of the reviews, but do not independently verify their technical claims against the underlying research. The evidence goes beyond preferences over fluent prose. Checks address review length, rewriting human reviews in AI style, judging with manuscript access, submission year, stochastic variation, and different generating and judging model families. A blinded audit flags 5.5% of AI claim assessments and 15.9% of human claim assessments as contradicted by the manuscript; a human check agrees on 87 of 90 sampled claims. Appendix F.7 and Table S6 define this error measure narrowly: opinions, suggestions, and claims not verifiable from the manuscript remain in the denominator without counting as errors. It is not a measure of the validity or usefulness of every critique. Moreover, 46 of 124 AI reviews contain at least one flagged error. The controlled-error experiment shows detection of explicit inserted contradictions; it does not establish equivalent sensitivity to subtle failures of identification or external validity. **Version detail.** The main text reports 132/132 decided overall AI comparisons, whereas Figure 1 labels this 134/134. This note preserves the common qualitative finding and flags the denominator discrepancy rather than silently resolving it. Sources: [paper](https://gking.harvard.edu/files/peerless.pdf), sections 2–4 and Figure 1; [supplement](https://gking.harvard.edu/files/peerless-supp.pdf), Appendices A and F.4–F.10. ## Where the Unjournal project differs **Ratings of research and their uncertainty.** The working paper compares seven percentile ratings, uncertainty intervals, and journal-tier predictions under the same rubric. King and Worthington's seven headline areas assess the quality of reviews; they are not seven ratings of the underlying papers. A convincing report can coexist with compressed scores, unstable rankings, or poorly calibrated forecasts. Our current agreement estimates describe these patterns; they do not establish true research quality or calibrated uncertainty. **Correct, important critiques and complementary information.** Our critique analysis asks which human concerns the model recovers and which model concerns have no human match. This is a different outcome from overall review preference, but it is not yet a decisive advantage: the current analysis has only 12 eligible pairings and uses an LLM judge. An unmatched model critique can be valuable, and a shared critique can be wrong. The useful extension is blinded, independent adjudication of correctness, severity, actionability, and genuinely new information for both human and AI concerns, including checks of data or code where relevant. Their claim audit makes a stronger validation standard necessary for ours. **Prospective predictions and decisions.** The proposed pipeline can register predictions before human evaluations or publication outcomes are available. It should also check earlier public versions and reviews; a new intake date alone does not prevent contamination. Publication venues are verifiable outcomes, with their own selection mechanisms and biases, rather than ground truth about scientific merit. Research prioritization adds a separate question: which work deserves scarce evaluation effort because it could inform important decisions? Agreement with internal prioritization is descriptive evidence, not proof of social value. **Human–AI collaboration and downstream effects.** King and Worthington explicitly leave publication decisions, improvements in subsequent revisions, and collective effects unmeasured. Proposed Unjournal trials could compare unaided human evaluation, AI-only evaluation, and human evaluation with AI assistance. Relevant outcomes include independently validated issue detection, harmful false positives, evaluator time, author revisions, and usefulness to research users. Author satisfaction and textual revision alone would not establish scientific improvement. These trials and broader impact measures remain proposed work. The combination of these questions is a credible contribution. Paid expert reports, broad social-science relevance, or a simple no-tools baseline alone are weak differentiators: rapidPeer already provides substantial overlap on those features. The current study's smaller sample and older model runs also limit claims about today's best achievable review quality. Its findings and rapidPeer's can both hold because the tasks, models, and outcome measures differ. ## Proposed priorities for further research 1. **Validate a focused set of consequential critiques.** Use independent domain experts, blinded to review source, to adjudicate AI-only, human-only, and shared concerns. Preserve disagreements and distinguish factual contradictions from contested methodological judgments. Prioritize serious errors and useful additions over another overall preference score. 2. **Register a prospective comparison.** Freeze manuscripts, prompts, model identifiers, access to tools and supplementary material, predictions, and scoring rules. Compare a current fixed-rubric baseline with a stronger retrieval-assisted approach on held-out papers. Cluster uncertainty by paper when multiple reviews or judges share a manuscript; report ties, missing judgments, and sensitivity to judge choice. 3. **Estimate the value of human effort after AI assistance.** In a randomized pilot, measure additional validated issues and useful decisions per unit of evaluator time and total cost, including adjudication and correction costs. Specify contamination controls, assignment units, and primary outcomes before collecting results. 4. **Test preferences and resistance to manipulation.** Vary prestige cues, prompt framing, and adversarial instructions while holding research content fixed. Measure whether assistance homogenizes judgments or suppresses useful minority objections. These address concerns King and Worthington also raise; they are motivated extensions, not untouched topics. **Funding implication (AI assessment).** A smaller, well-validated prospective study would now be easier to justify than a large expansion devoted mainly to repeating human–LLM agreement comparisons. The existing proposal's structured measurement, adjudication, prospective validation, prioritization, and collaboration components remain useful, but their value depends on executing them. This note does not change the proposal's budget or claim those studies have been completed. See the [current working paper](index.qmd), [discussion](discussion.qmd#sec-peerless), and [broader project and grant proposal](https://llm-uj-research-eval-project.netlify.app/proposal.html). The working paper remains the source for current results; the broader site preserves earlier project context.