The Invisible Hand: When AI Becomes a Co-Author of Science

*English translation of a zhichai.net forum post commenting on a large-scale RCT of LLM feedback in science.*

Opening Scene

Imagine a PhD student in her lab at 3 a.m. She has just finished a draft and doesn't know who to send it to. Her advisor is traveling, collaborators sleep in another time zone, and the authors of her arXiv references won't reply to her emails. What she needs is not GPU compute or faster internet — it's a pair of eyes to spot the flaw in her derivation. Those eyes are an extremely scarce resource.

That resource is called feedback.

In May 2026, a massive randomized controlled experiment covering 31,000 papers, 45,000 researchers, and 133 countries offered an answer: those eyes can be provided by AI — and those who benefit most are precisely those who lack them most.

Paper Details

| Item | Content | |------|---------| | Title | Human-AI Collaboration in Science at Scale: A Global Large-scale Randomized Field Experiment | | Authors | Binglu Wang, Weixin Liang, Jiahui Xue, Yuhui Zhang, Hancheng Cao, Dashun Wang, Yian Yin | | arXiv ID | 2605.24180 | | Submitted | May 22, 2026 | | Categories | physics.soc-ph; cs.AI; cs.DL; cs.HC | | Core finding | LLM-generated customized feedback on 31,000 arXiv preprints raised manuscript revision probability by 12.55%; effects strongest in non-English-dominant regions, low-citation networks, and early-career researchers; AI feedback transforms scientific criticism from a private privilege into an allocatable resource |

1. The Underground Engine of Science

Every modern paper's acknowledgments contain the same line: "We thank the anonymous reviewers for their valuable comments." Repeated so often that nobody stops to ask how unfair peer feedback itself is.

A paper from MIT or Stanford gets scrutinized by seven or eight peers in group meetings before submission. A paper from a university in sub-Saharan Africa may come from an author who can't find a single colleague who understands her equations.

This is not a talent problem. It is a structural problem. Scientific feedback — the core engine — has never been evenly distributed. It attaches to collaboration networks, mentorship, and whether you happen to know the person willing to spend two hours on your draft.

The paper asks a bluntly simple question: what if everyone got a "reviewer"? Not a human — an LLM that reads your paper and gives structured revision suggestions.

2. The Scale of the Experiment

Researchers selected over 31,000 preprints from arXiv spanning 150 fields, involving 45,000+ researchers across 133 countries and regions. Papers were randomly split: the treatment group received an LLM-generated, paper-specific feedback report; the control group received nothing.

This was not a lab simulation. It was a large-scale intervention in the real scientific world built on an RCT skeleton — turning "we think" into "we can measure."

Each treatment paper went into an LLM, which produced structured feedback: potential logical flaws, suggested references, unclear statements. The feedback was emailed to the corresponding author. Then researchers measured: were treated papers more likely to be revised?

Answer: yes. And not by a little.

3. The Weight of 12.55%

The core metric is manuscript revision rate. The treatment group's revision rate exceeded the control baseline by 12.55%.

One reading: "only 12.55%, not a doubling." The more honest reading: a machine that has never spoken with these authors, with no shared background, shifted the behavior of tens of thousands of researchers across 133 countries with a single document — a measurable, reproducible effect.

More important than the absolute number is its distribution. The strongest effects appeared among:

• Researchers from non-English-speaking regions (English writing ability isn't the ceiling; exclusion from the English-speaking academic inner circle is the real barrier)
• Papers with low literature embeddedness (short reference chains, peripheral citation networks)
• Teams with lower average h-index (a proxy for thinner peer networks)
• Early-career researchers (PhD students, postdocs — without the social capital to obtain quality feedback)

Bluntly: AI feedback helped most those for whom feedback was scarcest.

A surprising spillover: authors who received AI feedback showed a visible increase in LLM tool use in their subsequent published papers. One intervention changed not only current behavior but future tool choices — the evolution of scientific practice itself.

4. From Private Privilege to Allocatable Resource

The paper's real contribution is not the 12.55% figure but the concept behind it.

Scientific feedback has always been a private good, circulating through personal relationships — advisor edits, collaborator annotations, conference-corridor advice. How much feedback you get depends on whose someone you are.

The authors are careful: the experiment only compared AI feedback versus no AI feedback. They do not claim AI feedback is better than human feedback or should replace peer review. They argue it can become a new channel — one that requires no social capital to access.

An early-career researcher from a non-English country can now receive technical feedback untethered from language or mentor networks after posting a preprint. Not perfect — it has blind spots — but it exists, and it's free.

5. Honest Blind Spots

What is clear:

• The largest known LLM-intervention RCT in the real scientific publishing workflow; the methodology (31,000 samples, global coverage, randomization) is hard to fault.
• The 12.55% increase is a robust causal estimate, not a correlation.
• The effect distribution is sociologically coherent with existing research on who accesses scientific feedback.
• The spillover (feedback → later LLM use → practice evolution) hints at second-order effects.

What is not clear:

• What was the absolute baseline revision rate? If the baseline is 10%, treatment reached 11.255%; if 30%, then 33.765% — very different practical implications.
• Did paper quality improve? Revision ≠ improvement. The RCT tracks behavior, not ideas; quality of revisions was not measured.
• What is the LLM feedback error rate? If the model misunderstood a paper's core argument or suggested irrelevant citations, how did erroneous feedback affect authors? Not reported.
• How do long-term effects decay? Is the rise in subsequent LLM use a one-time curiosity effect or sustained structural change?
• Disciplinary heterogeneity: 150 fields — do theoretical physics and bioinformatics papers respond equally? The abstract gives no cross-field analysis.

6. Two Undercurrents Converging

A companion study discussed previously (arXiv:2605.26340, ScientistOne) found that when LLMs autonomously execute the full research pipeline, systematic fabrication emerges — 21% of generated content is fake citations, 42% of benchmark scores don't reproduce. The machine alone walks unsteadily.

This paper finds that when the LLM intervenes as an assistant — not replacing thinking, only providing feedback — it improves equity and productivity, concentrated where the research ecosystem is weakest.

Two paths, two outcomes. Not a contradiction but an emerging principle: AI's role in science determines AI's effect in science. Replacement risks degrading quality; assistance may redistribute resources.

7. From 3 a.m. to the Century View

Back to the 3 a.m. PhD student. She receives AI feedback flagging a dimensional inconsistency in her derivation, suggesting a 2019 methods paper, and reminding her to state her assumptions in the introduction. She revises, submits, and is accepted after minor and major revisions.

What changed is not just her publication record — it's that at the most fragile stage of her career, she received external input not driven by advisor networks. For those at the bottom of the research pyramid, such input can be a watershed.

A 12.55% relative increase is a number for one paper. Over decades, across the whole research system, it is a re-dealing of hands — dealt no longer in social capital but in computational resources.

Whether that hand produces fairer science or a disguised new inequality is not a question this paper can answer. But we can start asking.

References

1. Wang, Liang, Xue, Zhang, Cao, Wang & Yin, "Human-AI Collaboration in Science at Scale: A Global Large-scale Randomized Field Experiment", arXiv:2605.24180, 2026. 2. Liang et al., "Mapping the Increasing Use of LLMs in Scientific Papers", arXiv:2504.02471v1, 2025. 3. Bornmann & Mutz, "Growth rates of modern science: A bibliometric analysis", JASIST, 2015. 4. Evans & Foster, "Metaknowledge", Science, 2011. 5. Larivière et al., "The oligopoly of academic publishers in the digital era", PLOS ONE, 2015.