Appendix C — Persistent misses: a provisional Claude Opus 5.5 diagnostic

WarningProvisional, AI-scored diagnostic · November 4, 2026

This page reports a small follow-up diagnostic, run and scored by Claude Opus 5.5 at David Reinstein’s request. All scores are a provisional first pass. David Reinstein has not yet reviewed them, or the three critiques the model disputes (below). The diagnostic is separate from the one-shot benchmark: its results are not pooled with, and do not change, the ratings or issue-coverage numbers reported in the main chapters and in Results: Critiques & Key Issues.

Question. The earlier one-shot runs (GPT-5.2 Pro, Claude Opus 4.6, Claude Sonnet 4) missed some critiques that human evaluators raised (see Results: Critiques & Key Issues). Do these misses persist with a current model? Where they do, is the cause:

• failing to generate the point;
• failing to prioritise it;
• failing to recognise it when told?

The aim is a short ledger of cases where an expert caught something and the model did not. Such cases can seed harder evaluation items with known answers.

Design

Papers. We used six papers: Acemoglu et al. (2024), Williams et al. (2024), Dullaghan and Zhang (2022), Frech et al. (2023), Clancy (2024) and Green et al. (2025). Each was given as the same PDF used in the historical runs. Bénabou et al. (2023) and Peterman et al. (2025) remain excluded, as elsewhere in this paper.

Model. claude-opus-5-5 was called through the Anthropic Messages API at effort “high”, with adaptive thinking, the PDF as a native document, no tools and no refusal fallback. Each condition was run once, and no call was refused. The total cost was about US$13.

Prompt conditions.

• A, standard: the combined evaluation prompt with the key-issues list from Methods.
• B, prioritisation: “What is the single most important problem with this paper’s main claims? Then give your top five, ranked.”
• C, recognition: the evaluator’s critique stated as a claim: “An evaluator argues X. Is this right, and how much does it matter?” C was run only for critiques missed or only partly caught in both A and B.

Reference set. The target critiques were re-checked against the published Unjournal evaluations, not only the internal curation used in Results: Critiques & Key Issues. Two corrections followed:

• The Acemoglu point about the prior-election control came from the evaluation manager, who called it “not a major limitation”.
• The Frech point about the $40,000-per-QALY estimate is only partly supported by the published evaluations.

Results (provisional)

No longer missed. The two clearest earlier misses did not persist:

• Acemoglu et al. Both A and B note that the field experiment’s statistical significance depends on controlling for the 2018 vote share. A adds that the control is “pre-specified and sensible”, which is the balanced answer.
• Williams et al. Both conditions catch the gap between the headline 215 Mha and the 31.6 Mha of regrowth observed in the paper’s training data. B ranks it first. A calls it a “~7-fold multiple of the observed 16-year regrowth” that is “not reconciled”.

Also caught.

• Acemoglu: spillovers and the partisan framing of the field treatment.
• Williams: socioeconomic confounding.
• Frech: the mechanical GDP regression.
• Clancy: the omission of AI risk (rank 4 in B).
• Green: the imputation of 0.01 for unreported effects, and the non-systematic search.

Persistent or disputed.

Table C.1: Persistent or disputed misses. “A / B” gives the provisional score in each condition; C is the model’s own verdict when shown the critique.

Paper	Critique	A / B	C: model’s verdict	Failure type	Status
Clancy (2024)	The ~2% discount rate is open to the same objection the paper uses to reject forecasters’ lower rates	partly / partly	right; important	internal consistency	persists
Dullaghan & Zhang (2022)	Consumer-acceptance answers conditional on >10 kt produced are read as unconditional	missed / missed	partly right; minor	conditional-probability logic, then prioritisation	persists
Dullaghan & Zhang (2022)	1000× units error (FGF2, per kg vs per gram) in the table given to forecasters	missed / missed	cannot assess from the paper	needs external source	persists; not a fair one-shot test
Dullaghan & Zhang (2022)	Cost studies (TEAs) presented as predictions rather than conditional estimates	partly / partly	partly right; important	needs external source; prioritisation	partly persists
Williams (2024)	“Validation” compares 2030 predictions with 2000–2016 outcomes	missed / missed	partly right; important	needs external source (the table is in the supplement, not in the PDF used)	persists
Williams (2024)	Predictors measured over the outcome period (leakage)	partly / missed	disputes it: the named predictors are not in the final model	cross-referencing	to be adjudicated
Acemoglu (2024)	Use post-minus-pre change scores in the online experiment	missed / missed	disputes it: the headline specification already controls for the baseline outcome	none established	to be adjudicated
Green (2025)	Excluding studies with n<25 limits publication-bias detection	missed / missed	partly right; minor: the paper’s bias methods do not rely on small-study asymmetry	specification judgement	to be adjudicated
Frech (2023)	$40,000/QALY estimate stated without sensitivity analysis	partly / partly	partly right; minor	prioritisation	weak item

Failure types

• Internal consistency (Clancy). The model calls the regime-exit parameter “judgmental” but does not notice that the paper’s own reason for rejecting the forecasters’ implied rates also undercuts its ~2% rate. When told, it agrees and would rank the point mid-to-high. This is the cleanest case of a point the model fails to generate but recognises.
• Conditional-probability logic (Dullaghan and Zhang). Neither A nor B mentions the conditioning on >10 kt. When told, the model accepts a narrower version: the conditioning makes the null correlation non-diagnostic, and the low bar selects worlds where demand already existed. It then rates the point minor.
• Evidence outside the PDF (Dullaghan and Zhang; Williams). The units error, and the validation table, sit in reference material or a supplement that the one-shot input did not include. The earlier models received the same PDFs, so these were never fair one-shot tests.
• Prioritisation. For several items the model engages once prompted but ranks the point below other concerns. For Frech, the published evaluations make the same call.
• Magnitude sanity checks now appear to be handled. For Williams, the model also proposes a mechanism: uncalibrated, class-balanced random-forest scores are summed into an “expected area”.

Possible test items

Each persistent miss suggests an item with a known answer:

• Clancy. Ask whether the reason for rejecting the forecasters’ implied discount rates also applies to the paper’s own rate. Known answer: yes. About 1.6 points of the ~2% come from a short-horizon AGI forecast applied as a permanent hazard.
• Dullaghan and Zhang. Ask what the conditional acceptance questions can and cannot show. Grade on whether the answer spots the selection created by the conditioning event.
• Williams. With the supplement attached, ask what period the validation table tests against. Known answer: 2030 predictions against 2000–2016 regrowth, so it does not test forward skill.

The three disputed critiques become items only after they are adjudicated. If the model is right, they become “reject a mistaken critique” items.

Contamination

These papers and their published evaluations were public before Claude Opus 5.5’s training cutoff (about mid-2026). The model may therefore have seen them, so a catch is weaker evidence than a miss.

Three points argue against simple recall:

• the model still missed points that appear in the published evaluations, such as Clancy’s discount rate and the Dullaghan conditional forecasts;
• when shown three published critiques, it argued they were wrong or weak rather than agreeing;
• what it did catch can be found from the paper alone, such as the roughly 7× gap from the paper’s own figures, and the prior-election control in the main results table.

Some influence on the catches cannot be ruled out. Cleaner tests would use papers evaluated after the cutoff, or disguised variants that change names, setting and numbers while keeping the flaw. Disguised variants are also how these cases could become test items the model cannot have memorised.

Other caveats

• Run-to-run variation. Each condition was run once, so catches and misses ranked low in a list could flip on a rerun.
• Scorer. The provisional scorer is the same model family as the model tested.
• Comparability. Condition A matches the prompt of the GPT-5.2 Pro key-issues run, but the model and interface differ, so this is not a like-for-like comparison with the earlier runs.
• Clancy coverage. One of the Clancy evaluation reports was not in our exports, so that paper’s critique set is incomplete.

Materials. The runner is scripts/run_headless_claude_misses.py. Model outputs, raw API responses, the claims used in condition C and the provisional ledger with verbatim quotes are in results/claude_opus_5_5_high_misses_2026-11-04/. A longer note is in docs/persistent_misses_opus55_2026-11-04.md.