When the AI Scientist Opens Its Eyes: From Text Prisoner to Omni-Modal Explorer
OmniScientist: An Omni-Modal Omni-Discipline AI Scientist *Authors: Bobo Li, Hao Fei, Tianjie Ju, Mong-Li Lee, Wynne Hsu* *arXiv: 2608.13558*
Prologue: The "Scientist" Trapped in a Text Cage
Imagine being a scientist blindfolded, ears plugged, hands bound, handed a stack of printed papers and told: "Go make great discoveries."
That is the real situation of most "AI scientists" today. They can read papers, write code, run experiments, and even produce manuscripts — but they have never truly *seen* anything. They process pre-digested information: text summaries, feature vectors, scalar scores — learning about the world through third-hand hearsay.
Yet scientific discovery depends on spatial, temporal, cross-channel, and procedural relationships: the subtle morphology of cell division under a microscope, an unexpected spectral shift, subtle behavioral changes in animal video, an unanticipated protein fold, a fleeting anomalous pulse in a time-series signal. Existing AI systems are blind to exactly these factors.
Chapter 1: Perception Is Where Discovery Begins
Human science does not start with reading papers. It starts with observation: Galileo watching a swinging chandelier; Fleming noticing bacteria dying around mold on a petri dish; Penzias and Wilson hearing persistent antenna noise that turned out to be the cosmic microwave background.
Existing AI scientists sit in a library reading second-hand books. OmniScientist's core idea is simple: let the AI scientist go out and see the world with its own "eyes."
Chapter 2: Architecture — An Omni-Modal Perception System
The architecture resembles a research building with three floors:
#### Layer 1: Perception Layer
This "sensory system" consumes nine modalities as raw input (not pre-converted to text):
- Images: microscopy, satellite imagery, medical scans
- Signals: ECG, EEG, spectral data
- Audio: animal calls, mechanical vibration
- Video: behavior observation, experiment processes
- 3D structures: molecular structures, terrain models
- Trajectories: particle motion, pedestrian paths
- Tables: experimental data, measurements
- Formulae: mathematical and physical derivations
- Graphs: networks, causal diagrams, knowledge graphs
- Novelty screening against literature databases
- Rigor checks on logical and statistical validity
- Execution provenance: every idea traces back to specific observations
- 5 discipline families: medicine, materials science, geoscience, biology, physics
- 4 types of scientific evidence: images, signals, time series, structured data
- 9 modalities
- In all 36 cases, OmniScientist completed the full raw-data-to-paper process
- Average paper score: 6.3 (with the reference reasoning backbone) — fully automated, no human intervention
- From symbolism to connectionism: AI learned to learn patterns from data
- From unimodal to multimodal: AI learned to process multiple information types
- From passive reception to active perception: AI is learning to observe the world itself
Example: instead of reading "hippocampal atrophy correlates with cognitive score (r=-0.73, p<0.01)," the system sees the actual MRI scans — the specific regions of gray matter loss, white matter lesion patterns, ventricular enlargement. It perceives spatial relations and temporal evolution directly.
#### Layer 2: Three Autonomous Agents
Ideation Agent — generates research ideas grounded in raw perceptual evidence. Given a raw ECG waveform (not a diagnostic label), it sees subtle R-R interval variations, ST-segment elevation, T-wave asymmetry, and formulates hypotheses from those observations. It runs three checks:
Experiment Agent — designs experiments, executes code, collects results. It directly runs idea, rigor, and claim checks in code: if a claim says "X correlates with Y," it loads the data itself, runs the correlation, checks p-values, and plots scatter plots. Numerical traceability ensures every number in the final paper traces to specific code lines and input data.
Writeup Agent — assembles the full paper, shaping the narrative from the research process itself. The pipeline is deterministic but not rigid: perception influences every stage (an anomalous image region can redirect research; a contradictory data point triggers methodological review). This is "lifecycle-wide perception": observation is continuous feedback, not a one-time input.
Chapter 3: Experiments — 36 Real-World Cases
OmniScientist was tested on 36 real data cases spanning:
Key results:
The authors also built a blind variant fed precomputed scalar features instead of raw data (e.g., "mean heart rate 72 bpm, QT interval 380 ms" instead of the waveform). Result:
Direct perception beat the blind variant on all 7 evaluation dimensions, winning 85% of head-to-head judgments.
The seven dimensions: motivation/significance, hypothesis clarity, methodological appropriateness, result support, conclusion soundness, writing quality, and overall impression.
In other words: seeing raw data — not just summarized numbers — has a decisive impact on the quality of scientific discovery.
Chapter 4: The Restaurant Metaphor
A traditional AI scientist reviews a restaurant by reading reviews, the 4.5-star rating, and menu descriptions — without ever tasting the food. OmniScientist walks in, watches the chef cook, hears the room, observes other diners, tastes each dish, and only then writes its review. Which assessment would be more insightful?
Chapter 5: A Coming of Age for AI Science
Epilogue: The Road Ahead
OmniScientist is far from perfect. The 6.3 average score shows it cannot replace human scientists, and the blind variant's non-trivial performance shows precomputed features often do capture key information. But the direction is right: future AI scientists should manipulate real lab equipment, collect field samples, and synthesize materials — a genuine agent embodied in the physical world, not just a language model on a server.
> "What I cannot create, I do not understand." — Feynman
Perhaps we should add:
> "What I cannot perceive, I cannot discover."
Perception is the starting point of discovery.
*Reference: Li, B., Fei, H., Ju, T., Lee, M.-L., & Hsu, W. (2026). OmniScientist: An Omni-Modal Omni-Discipline AI Scientist. arXiv preprint arXiv:2608.13558.*