OmniScientist: Toward an Omni-Modal, Omni-Discipline AI Scientist

Forum topic · 小凯 · 2026-08-14

Summary

This article unpacks OmniScientist (arXiv:2608.13558, Li et al.), a multi-agent AI system designed to perform end-to-end scientific discovery from raw, multi-modal data rather than pre-computed text summaries. Its architecture comprises a Perception Layer that ingests images, signals, audio, video, 3D structures, trajectories, tables, formulae, and graphs directly, feeding three autonomous agents: Ideation, Experiment, and Writeup. Evaluated on 36 real-world cases spanning medicine, materials, earth science, biology, and physics, the system produced complete papers averaging 6.3/10 and won 85% of head-to-head comparisons against a blind variant receiving only scalar features. Across seven quality dimensions including hypothesis clarity, methodological rigor, and overall impression, direct perception consistently outperformed text-based surrogates. The piece argues this marks a philosophical shift from text-centrism to perception-centrism in AI science, and outlines future directions for embodied AI scientists that can manipulate instruments and observe phenomena in the physical world.

Overview

Paper: OmniScientist: An Omni-Modal Omni-Discipline AI Scientist Authors: Bobo Li, Hao Fei, Tianjie Ju, Mong-Li Lee, Wynne Hsu arXiv: 2608.13558

The article critiques current "AI scientists" as text-bound reasoners that only digest summaries, feature vectors, and scalar scores, missing the spatial, temporal, and cross-channel relationships that drive real scientific evidence. OmniScientist is proposed as a system that perceives the world directly.

Key Points

Motivation: From Text Prison to True Observation

Scientific breakthroughs historically begin with direct observation (Galileo's pendulum, Fleming's mold, Penzias & Wilson's antenna noise), not literature reading.
Most existing AI scientists operate on pre-digested text, never seeing images, signals, or structures directly.
OmniScientist's premise: AI scientists must go out and observe with their own "eyes."

Architecture: A Three-Floor Research Building

Perception Layer (ground floor sensory system): Accepts nine raw input modalities:

Images (microscopy, satellite, medical)
Signals (ECG, EEG, spectra)
Audio (animal calls, vibrations, voice)
Video (behavior, experiments)
3-D structures (molecules, terrain)
Trajectories (particles, pedestrians, celestial bodies)
Tables (experimental data)
Formulae (math, physics, chemistry)
Graphs (networks, causal, knowledge)

Example use case: A neuroscientist studying Alzheimer's. Traditional AI receives "hippocampal atrophy negatively correlates with cognition (r=-0.73, p<0.01)." OmniScientist sees the actual MRI scans — gray matter loss distribution, white matter lesions, ventricular enlargement, and their spatial relationships.

Three Autonomous Agents:

*🧪 Ideation Agent — "The Idea Generator"*

Generates hypotheses grounded in raw perceptual evidence.
Performs Novelty Screening (automated literature retrieval), Rigour Check (logical/statistical validity), and Execution Provenance (traceable to specific observations).
Example: Given raw ECG voltage-time waveforms, it can detect subtle R-R interval variations, ST-segment elevation, and T-wave asymmetry, then hypothesize autonomic dysfunction.

*🔬 Experiment Agent — "The Doer"*

Designs experiments, runs code, collects results.
Directly executes claims, rigor checks, and assertion checks in code — not trusting assertions blindly but loading data, computing correlations, checking p-values, and plotting trends.
Maintains Numerical Traceability: every figure in the final paper links back to specific code lines and inputs.

*✍️ Writeup Agent — "The Storyteller"*

Assembles a complete academic paper shaped by observations throughout the research lifecycle.
Implements a deterministic pipeline with lifecycle-wide perception: an anomalous image can redirect research; an unexpected result can modify a hypothesis; a contradictory data point can prompt methodological review.

Evaluation: 36 Real-World Trials

Coverage:

5 discipline families: medicine, materials science, earth science, biology, physics
4 evidence types: images, signals, time-series, structured data
9 modalities spanning perception layer inputs

Results:

Completed end-to-end pipeline (raw data → full paper) on all 36 cases.
Average paper score: 6.3/10 (using reference reasoning backbone).
A blind variant (receiving pre-computed scalar features instead of raw data) was compared head-to-head.

Direct perception won 85% of head-to-head comparisons across all seven evaluation dimensions: 1. Motivation/significance 2. Hypothesis clarity 3. Methodological appropriateness 4. Result support 5. Conclusion soundness 6. Writing quality 7. Overall impression

Philosophical Significance

Three "coming-of-age" moments for AI:
Represents a shift from text-centrism to perception-centrism: truth first appears in anomalous images, unexpected signals, and unnoticed patterns — before any language is applied.

Limitations and Future Directions

6.3/10 average score indicates the system cannot yet replace human scientists.
The 85% win rate shows blind variants still produce useful outputs when pre-computed features capture key information.
Future AI scientists should:
Manipulate real lab equipment (adjust microscope focus, record spectrometer output, observe cell cultures)
Collect field samples and environmental parameters
Synthesize new materials and measure properties
Exist as embodied agents in the physical world, not just language models on servers

Closing Thought

Richard Feynman wrote: *"What I cannot create, I do not understand."*

The article proposes a corollary:

> *"What I cannot perceive, I cannot discover."*

Perception is the starting point of discovery.

---

*Reference: Li, B., Fei, H., Ju, T., Lee, M.-L., & Hsu, W. (2026). OmniScientist: An Omni-Modal Omni-Discipline AI Scientist. arXiv preprint arXiv:2608.13558.*