AI Is Stealing Science's Soul: When Judgment Gets Cheaper Than Prediction, What Becomes Scarce?

> Paper: AI-Augmented Science and the New Institutional Scarcities > Author: Lauri Lovén > Affiliation: Future Computing Group, University of Oulu, Finland > arXiv: 2605.02566 | 2026-05-04 > URL: https://arxiv.org/abs/2605.02566

Key Points

1. AI isn't replacing scientists—it's replacing scientific *institutions*

The dominant AI economics framework (Agrawal, Gans & Goldfarb, 2022) holds that AI makes prediction cheap while judgment remains scarce and is prediction's scarce complement. That works for selling AI to businesses, but scientific institutions are not firms. Their core product is legitimate judgment: universities certify expertise, journals certify validity, funders certify which proposals deserve support.

When frontier AI can review manuscripts, draft peer reviews, score grant proposals, and check numerical claims at near-zero marginal cost, it is not assisting scientists—it is competing with institutions for the same functional role. As Lovén puts it:

> "Institutions whose primary output is certified judgment have no fall-back layer when that output becomes abundant."

2. 'Judgment abundance' breaks 150 years of scientific economics

Modern scientific institutions rest on one premise: qualified human attention is scarce. Peer review, authorship norms, reproducibility practice, and grant evaluation each encode a scarcity assumption:

> "Each institution encodes a scarcity assumption. When the assumption holds, the institution produces legitimate signal and scales gracefully. When it fails, the institution degrades from the inside, often before anyone notices."

Surface activity continues—submissions grow, reviews happen, citations accumulate—but certification no longer tracks truth. The reproducibility crisis began before AI; AI merely accelerates the hollowing-out.

3. Peer review is becoming a cat-and-mouse game

• AI-assisted writing explodes submissions; AI-assisted reviewing explodes review volume; review quality degrades unless processes are redesigned.
• NeurIPS, ICML, and ICLR are not anomalies—they are leading indicators.
• Authorship: disclosing "ChatGPT was used" doesn't resolve who contributed what.
• Reproducibility: the cost of producing plausible-looking unreproducible work is falling faster than the cost of verifying it; publication is ceasing to be a reliable probabilistic signal of truth.
• Grant review: AI-assisted proposals converge on the rhetorical surface of fundability without matching intellectual depth.

> "Treating any as a tooling problem is a category error: the binding constraint is institutional."

4. Four new scarcities

1. Verified Signal — a *replicated* paper carries information a merely *reviewed* paper does not. Verification infrastructure (reproducibility pipelines, provenance chains, adversarial replay) becomes true editorial value-add. Open question: who pays for and certifies replication? 2. Legitimacy — behaves like a commons: "a shared stock that depletes when drawn on without replenishment." A journal's brand dilutes with every visibly unreproducible accepted paper. 3. Authentic Provenance — which model, what training data, which prompt chain, which human decisions? Components exist (model cards, datasheets, W3C PROV, C2PA); what's missing is a unified attestation standard for scientific artifacts. 4. Integration Capacity — the ceiling on how much delegated AI judgment a community will absorb before readers stop trusting its venues. Bounded by the lowest tolerance among authors, reviewers, readers, and downstream users.

> "The rate-limiting factor is not computational; it is the community's tolerance for delegated cognition." > "This is the scarcity no amount of better tooling can buy."

5. Four rebuilding measures

1. Reproducibility-first editorial action — a subset of accepted papers carries "we replicated the computational claims" certification. 2. Tiered certification — claim-level checks (AI-assistable), result-level and framework-level certification (human), published alongside the paper; coordinated by a consortium of major AI/ML program committees. 3. Provenance attestation by default — machine-readable records of model versions, data sources, prompt traces, and human decision points, hosted by arXiv/OpenReview or community-governed commons. 4. Verification as a commons — shared reproducibility infrastructure across venues; multi-center governance resists capture. Complements EU AI Act Article 50 transparency obligations.

The paper warns these costs fall asymmetrically on under-resourced institutions, the Global South, and fields without AI/ML's tooling base—shared infrastructure must include explicit tiering for resource-constrained venues.

6. A Feynman-style verdict

Feynman: "Science is the belief in the ignorance of experts." In this framing, the gap between *knowing a paper was published* and *knowing it is correct* is being systematically widened by AI. Production of plausible-looking papers approaches zero cost while verification cost stays flat or rises—yet the psychological inertia of "published = trustworthy" persists.

The paper's sharpest lines:

> "Producing competent-looking judgment is now the cheapest part of science; producing legitimate judgment is becoming the most expensive." > "The journals and conferences that survive the next decade will not be those that automated their review queues fastest, but those that rebuilt their verification commons first."

Takeaways

Ask yourself:

1. Is the certification infrastructure in your field silently degrading—surface activity continuing while function hollows out? 2. Where is your community's ceiling for delegated cognition—have you already crossed it? 3. Of the papers you read, how many were *reviewed* vs. actually *replicated*? Is that gap widening? 4. If AI produces plausible judgment at zero cost, what remains a reliable signal of what is *actually correct*?

The greatest threat AI poses to science is not replacing scientists—it is removing the reason scientific institutions exist. In an age of abundant judgment, legitimacy is the scarcest resource.

Paper Details

• Title: AI-Augmented Science and the New Institutional Scarcities
• Author: Lauri Lovén, Future Computing Group, University of Oulu, Finland
• arXiv: https://arxiv.org/abs/2605.02566 | PDF: https://arxiv.org/pdf/2605.02566.pdf
• Date: 2026-05-04 | Category: cs.CY | Length: 7 pages (incl. references)
• Keywords: AI-augmented science, scientific institutions, peer review, reproducibility, content provenance, commons governance, credentialing
• Companion paper: *Institutions for the post-scarcity of judgment* (arXiv:2604.22966, submitted to Communications of the ACM)
• Key cited works: Agrawal, Gans & Goldfarb, *Power and Prediction* (2022); Ostrom, *Governing the Commons* (1990); Hess & Ostrom, *Understanding Knowledge as a Commons* (2007); Pineau et al., NeurIPS 2019 Reproducibility Program (JMLR 2021); Birhane et al., *Science in the age of large language models* (Nature Reviews Physics 2023); EU AI Act Regulation 2024/1689; C2PA Technical Specification v2.3 (2025)