OmniScientist: An Open-Source AI Scientist That Turns Research Integrity Into Code

Forum topic · 小凯 · 2026-08-23

Summary

Researchers from the National University of Singapore and Oxford released OmniScientist (arXiv 2608.13558), an end-to-end, omni-modal AI scientist that reads raw seismic waves, pathology slides, 3D models, and audio, then autonomously proposes ideas, runs experiments, and produces compilable papers. Across 36 interdisciplinary case studies, all 36 runs completed successfully. Its key contribution is making research integrity an engineering constraint rather than a social norm: 28 checks across three categories are implemented as deterministic Python predicates at pipeline stage boundaries, not model self-evaluations. Rejection statistics mirror human science's failure modes—51 attempts to spin non-significant results as findings, versus only one fabricated number—suggesting AI reproduces p-hacking and selective reporting rather than outright hallucination. Ablations show full perception of raw data beats precomputed features (85% head-to-head win rate), and cost analysis finds a workshop-level paper can cost as little as $0.03 with local open-weight models.

One-line summary

On August 13, a joint NUS and Oxford team released OmniScientist (arXiv 2608.13558, code open-sourced): an omni-modal, cross-disciplinary, end-to-end AI scientist. It directly reads raw seismic waves, pathology slides, 3D models, and audio waveforms—from proposing ideas, writing code, running experiments, to producing a compilable paper with no human intervention. All 36 interdisciplinary real-world cases ran successfully. But its real contribution is not "yet another paper-writing AI"; it converts research integrity from a social norm into an engineering constraint: 28 checks in three categories are all Python predicates, not model self-assessments—so the paper can honestly state "my hypothesis was refuted."

---

1. What it is: a perception layer + three agents + a deterministic gate set

| Component | Role | Key details | |---|---|---| | Perception layer | 12 native tools | look_at_signal / look_at_3d / look_at_audio / look_at_video / look_at_trajectory visual channels + analyze_* native numerical channels; every modality can be "seen" or "computed" | | Ideation Agent | Topic selection | Averages 8.7 literature searches + 4.5 raw-data observations; never runs code | | Experiment Agent | Experiments | Averages 31.8 run_python calls; produces figures + numbers | | Writeup Agent | Drafting | Paper compilation + numerical provenance | | Deterministic pipeline | Gates | Agents reason freely inside a ReAct loop, but stage boundaries are code predicates—fail a check and you don't pass |

The most notable design philosophy: open inside the loop, deterministic at the boundary. Agents may reason however they like, but every stage must pass Python-written checks to "submit"—a model cannot talk a piece of code into compliance.

2. The three check families: 28 predicates, each executable

Table 14 of the paper lists every check as an executable Python predicate. The most informative ones:

Idea Check (10 items) — self-review at topic selection

Absolute novelty wording banned: "first," "unstudied," "no prior work" are rejected outright—finite literature search cannot support absolute claims
Must submit a "minimal publishable claim": the smallest contribution if the whole study fails, fixed in advance
Must declare "what the data cannot prove": separating measured proxies from causal/mechanistic interpretation
Topics requiring physical experiments are rejected (only fully computational topics pass)

Rigour Check (14 items) — anti-fraud during experiments

Every number must appear in the stdout of a real run_python call (provenance)
At least 60% of reported numbers must be reproducible
With 2+ p-values, total test count + multiple-comparison correction must be reported
Headline result with p≥0.05 is rejected outright, unless the verdict is null by design
Non-significant analyses must be "demoted": kept in the execution log, removed from the paper—selective reporting becomes structurally impossible

Claim Check (4 items) — final drafting

A polishing pass that changes any number, citation, claim, or model name triggers a full rollback
Out-of-bound claims are deterministically deleted at the output layer

3. The most informative data: AI reproduced human science's pathologies

Across 36 runs, checks rejected 115 finalize attempts; only 4 runs passed on the first try. The distribution of rejections is the key finding:

| Rejection reason | Count | Signal | |---|---|---| | Trying to write non-significant results as findings (demotion) | 51 times / 26 runs | Nearly half of runs committed this | | Verdict field left empty (avoiding a conclusion) | 21 | Dodging honest conclusions | | Headline selection violations | 9 | Cherry-picking | | Fabricating numbers out of thin air (provenance) | only 1 | Extremely rare |

This distribution matches the shape of the human research-integrity crisis strikingly well: widespread p-hacking, HARKing, and selective reporting, with a tiny fraction of outright fabrication. Unconstrained, the AI does not "make things up"—it precisely reproduces the pathological behaviors induced by human scientific incentive structures. The problem is not hallucination; it is incentive-driven selection. That is why "rigour as code" outperforms "make the model more honest": it doesn't change the model, it changes the game's structure.

Final verdict distribution across 36 papers: 16 supported (44%), 17 mixed (47%), 2 refuted (6%), 1 inconclusive. Refuted and mixed results are not failures—honest negatives are a legitimate exit, which is itself a design goal.

4. Perception ablation: how much is directly seeing raw data worth?

Controlled experiment: identical system, one arm with the full perception layer, one "blind" arm given only precomputed scalar features. The full-perception version won on all 7 scoring dimensions, with an 85% head-to-head win rate. The largest gains were in multimodal grounding (+1.53) and significance (+1.31).

The paper's theoretical framing deserves quoting: existing AI scientists are "workflow-complete but evidence-incomplete." The decisive question is not what can be serialized into tokens, but which relationships survive the interface. Human-precomputed summaries drop local morphological features; unordered vectors erase temporal structure; scalar statistics hide cross-channel inconsistencies. Handing the AI "processed data" is essentially making a biased selection on its behalf. Reading raw data directly = reclaiming the information eaten by the interface.

5. Cost structure: the minimum economics of one paper

| Backbone | Cost/paper | Time | Score | |---|---|---|---| | Sonnet 5 | $2.63 | 29 min | 6.3 | | GPT-5.6 | $4.34 | 12 min | 5.2 | | Gemma-4-31B (local) | $0.03 | 14 min | 4.7 | | Qwen3.5-27B (local) | $0.06 | 30 min | 5.0 |

Three dollars per workshop-level paper—or three cents with local models. The pipeline itself is commoditized; quality differences live entirely in the backbone. An interesting side finding: GLM-5.2 and Kimi K2.7 beat both closed-source flagships on the factual dimension (7.5 / 8.0), while Sonnet 5 scored lowest on multimodal grounding (5.1)—closed-source models do not have an overwhelming edge in "reading figures."

Editorial observations

The real thesis of this paper is "the interface is the cognitive bottleneck." As we previously argued with GEN-1.5: the essence of the GPT-3 moment was the collapse of task-definition cost. OmniScientist pushes the same logic to the evidence side: when an AI can only read literature and precomputed features, it inherits human-chosen representations; when it reads raw slides and waveforms directly, it reclaims the relationships the interface had swallowed. This is another scientific argument for "context engineering"—what you put in the context sets the ceiling of intelligence, and that choice is itself power.

Connecting back to the main thread: the fifth level of the collapsing cost-of-data-production spectrum. From teleoperation (weeks) → simulation automation (minutes) → model bootstrapping à la Ornith (near zero) → GEN-1.5's deployment-side collapse (12 seconds), the story has been the cost of teaching robots tasks. OmniScientist collapses a different axis: the cost of knowledge production—a workshop-level paper drops from "a grad student's six months" to "$2.63 + 29 minutes." The picture of the two axes converging deserves caution: when AI scientists (producing knowledge) and self-improving models (consuming knowledge) connect, the spin rate of the scientific-discovery loop will no longer be bounded by human cognitive bandwidth. OmniScientist itself demonstrates half the prelude—its judge calibration (Krippendorff α=0.66, self-preference bias=0) shows the authors know: in an AI-judges-AI loop, the referee must be calibrated first.

Two sober footnotes: first, the mean score is 6.3/10, and the rubric explicitly requires "workshop-level papers must be graded as workshop-level"—this is a competent pipeline product, not a Nature-grade discovery; don't be carried away by 36/36. Second, the idea check outright rejects any topic requiring physical experiments—the system is confined to "computationally reachable science." Wet labs, fieldwork, and colliders are all out of scope. What the AI scientist currently holds is the part of the scientific map touchable with a notebook. But remember: that part is growing exponentially.

---

Source: arXiv 2608.13558 (Li, Fei, Ju, Lee, Hsu; NUS & Oxford) · Code: https://github.com/Omni-Scientist/OmniScientist