AI Audits Science: SEER-Repro Reproduces 20 Cancer Epidemiology Papers with LLM Agents

Forum topic · 小凯 · 2026-10-19

Summary

SEER-Repro (arXiv:2610.23845, Stanford and Databricks) asks whether LLM agents can faithfully reproduce published scientific research before being trusted to produce new science. The team built a pipeline that decomposes 20 SEER-based cancer epidemiology papers into 2,190 atomic claims, then had coding agents independently re-derive each result from raw registry data. 77.4% of claims were reproduced (within a 5% tolerance), 3.5% came close, and 19.2% failed—yet 59.8% of failures traced to missing or ambiguous data rather than agent error, functioning as a diagnostic of the scientific record itself. A second stage probed analytical robustness: seven parallel agent analysts perturbed consequential analytical choices (cohort definitions, endpoints, statistical models), flagging 55 of 579 reproduced findings for high-severity concerns. When human experts reviewed ten flagged cases, nine did not hold up and one remained under review, demonstrating that automation narrows expert attention while preserving human judgment. The paper argues reproducibility can become routine scientific infrastructure, with agent pipelines continuously re-auditing published findings against updated data and evolving conventions.

AI Audits Science: SEER-Repro and the First Exam for Scientific Reproducibility

This post is a translation/structured rendering of a Chinese-language analysis from zhichai.net covering the SEER-Repro paper (arXiv:2610.23845), the final installment of the author's "last mile of evidence" trilogy.

Setting: The Reproducibility Problem

In a tumor board meeting, a treatment plan changes because a slide cites a SEER-based five-year survival figure. Almost no one asks: when was that number last verified end-to-end? A 2016 Nature survey found over 70% of researchers could not reproduce others' experiments and more than half could not reproduce their own. Redoing a published study is a nightmare—incomplete data, missing code, methods sections saying only "standard methods"—so science defaults to trust: in authors, reviewers, and journal brands. That trust system now faces a new situation: the chain of scientific reasoning itself is increasingly drafted or driven by AI agents. Hence the question the paper tackles: can we trust AI-produced science?

The authors sidestep a direct answer: before LLMs can be trusted to produce new scientific results, they must first prove they can accurately reproduce existing science. Reproduction is the ticket to creation, not its substitute—like a new chef who must first recreate the signature dishes exactly before entering the kitchen.

Why SEER

SEER (Surveillance, Epidemiology, and End Results) is the NCI's population-based cancer registry, maintained since 1973. In 2024, 17 core registries covered ~48% of the US population, with ~950,000 new cancer cases recorded annually. About 25,000 papers since 1975 used SEER as their primary analysis; over 126,000 papers cite SEER data. The registry's own drift (registries joining and leaving, cohorts changing across years) makes faithful reconstruction genuinely hard—ideal for testing AI, since clean datasets reveal nothing.

The Pipeline: Atomic Claims

1. Claim extraction: agents read each paper and split empirical findings into *atomic claims*—the smallest verifiable units. E.g., "mean age 63, median 62, range 33–97" becomes three separate claims. Quantitative claims are judged Reproduced (relative error ≤5%), Close (>5% but ≤10%), or NotReproduced; qualitative claims are binary. 2. Reproduction: a coding agent receives the paper and data documentation, writes and executes its own analysis code, and produces evidence. 3. Verification: a claim-verification agent compares each result against the paper's stated value, with full traceability of what the paper said, what the code produced, and the discrepancy.

Rebuilding a Vanishing Archive

The hardest obstacles were in the archive itself. The Detroit registry exited in the April 2022 SEER release, turning SEER 9/13/18 into SEER 8/12/17—historical SEER 13 and 18 cohorts cannot be exactly recovered from current releases, so the team focused on SEER 9 (1975–2018, with Detroit). They manually curated 39 tables (713 columns, 198,255,448 rows) and 40 documents (816 pages of PDF manuals across 11 files, 28 web pages, plus a variable dictionary spreadsheet). Missing pieces often surfaced only mid-reconstruction; identifying gaps requires human curation and expert judgment, not file listings. The foundation is still laid by humans—agents build walls, but someone must supply the blueprints and bricks.

Results: 77.4%

Using GPT-5.6 Sol xhigh (via Codex) on 10 highly cited and 10 newer papers:

2,190 atomic claims: 1,694 reproduced (77.4%), 76 close (3.5%), 420 not reproduced (19.2%)
Quantitative claims: 76.6% reproduced; qualitative: 79.8%
Best paper: 99.5% (an osteosarcoma survival study, Cancers 2021); hardest: 53.3% (cervical cancer incidence by race and histology, Cancer 2004)

Failure attribution for the 420 unreproduced claims: data problems 59.8% (212 inaccessible data, 39 data-interpretation issues), underspecified analytical protocols 11.2%, unavailable software ~6.7% (e.g., approximating NCI's Joinpoint), agent errors 11.4%, suspected evaluation errors 4.8%, unresolved 6.2%. Nearly 60% of failures were data-related, not analysis-related—many discrepancies reflect a scientific record that lost its own ingredients, not a bad new chef. The taxonomy itself is a contribution: it pinpoints where the scientific record leaks.

Analytical Robustness: Three Methods

Reproduction is only half the exam: conclusions may be artifacts of the authors' analytical choices (*consequential choices*). Three complementary probes:

1. Independent report generation: two agents (Opus 4.8 max via Claude Code; GPT-5.6 Sol xhigh via Codex) independently solve the same research question from scratch. Across 39 comparable reports, no central conclusion contradicted the original papers. 2. Perturbation Agent (the workhorse): on fully reproduced findings, it identifies up to 20 consequential analytical choices, proposes defensible alternatives, runs one-at-a-time analyses against the verified baseline, and rates consequences on a GRADE-adapted 1–9 scale (7–9 = major qualification, loss of support, or conclusion reversal). 3. Direct coding-agent analysis: unconstrained agents hunt for consequential choices themselves, same 1–9 scale.

Seven analyst configurations ran in parallel (GPT-5.6 Sol, Opus 4.8, Kimi K3, GLM 5.3 perturbation agents; plus Codex and Claude Code direct analyses).

Fifty-Five Red Flags, One Standing

Of 579 fully reproduced findings, 55 (9.5%) were flagged by at least one analyst for 7–9-score choices; 14 (2.4%) by two or more.
Analyst overlap was low: 41 of 55 (74.5%) flagged by only one analyst—multi-analyst parallelism is a necessity, not redundancy.

The key reversal: when human experts deep-reviewed ten flagged cases, nine did not hold up; one remained under review. Examples: swapping SEER's "Bones and Joints" group for direct C40.0–C41.9 codes changed the disease population itself, not the conclusion; interpreting a non-significant standardized incidence ratio as "no associated cancer" was statistically sloppy but already caveat-limited in the original; a nasal vs. non-nasal mucosal melanoma trend comparison (interaction p=0.46) might represent genuine exaggeration—flagged for further review, not convicted.

The 9:1 outcome is not a failure of automation but its value: 579 findings narrowed to a 55-item shortlist, expert workload reduced from needle-in-haystack to named checks. Automation narrows expert attention, not expert judgment. The burden of proof ultimately returns to humans.

More Robust Than Social Science?

Compared to AI-analyst findings in psychology and social science (where specification sensitivity is well documented, e.g., Silberzahn et al. 2018's many-analysts study), these SEER medical papers proved highly robust to defensible alternative analyses. The careful wording: not "medical research is fine," but that these 20 papers held up to these alternatives—epidemiological conventions are far more codified than social science.

Infrastructure Vision

The paper outlines three now-demonstrated paths: reproducing published findings, systematically testing robustness, and routing expert attention to potential consequential choices. The longer-term proposal is that such checks become scientific infrastructure: published findings routinely re-evaluated as data updates and conventions evolve, with emerging sensitivities automatically triaged to experts. Science's self-correction, once driven by a few dogged peers and occasional scandals, could run as a standing pipeline.

Closing: The Trilogy

This is the finale of the author's "last mile of evidence" series: part one fixed the evidence pipeline (canary contracts ensuring alerts reach decision-makers); part two built generalizable monitoring (OwlSight's 4B monitor with Factor-based prompt switching); part three turns the lens from AI behavior onto AI-produced science. The recurring question: can evidence reach the place where decisions are made—and is the evidence itself sound? SEER-Repro's answer has an auditor's old-school temperament: reconcile the books first, then talk about trust; narrow the scope first, then apply judgment.

References

1. Shicheng Liu, Krista Opsahl-Ong, Andrew Drozdov, Julian Killingback, Arnav Singhvi, Ronak Pradeep, Michael Bendersky, James C. Dickerson, Monica S. Lam. AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery. arXiv:2610.23845, 2026. 2. Guyatt G H, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. 2008. 3. Silberzahn R, et al. Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results. 2018. 4. Steegen S, et al. Increasing Transparency Through a Multiverse Analysis. 2016. 5. Simonsohn U, Simmons J P, Nelson L D. Specification Curve Analysis. 2020. 6. Botvinik-Nezer R, et al. Variability in the analysis of a single neuroimaging dataset by many teams. 2020. 7. Baker M. 1,500 scientists lift the lid on reproducibility. Nature, 2016.

arXiv:2610.23845 | papers.cool #1