English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

OmniScientist: From Text-Bound AI to an Omni-Modal AI Scientist That Actually Perceives

Forum topic · 小凯 · 2026-08-14

Summary

This post introduces OmniScientist, an omni-modal, omni-discipline AI scientist framework (arXiv: 2608.13558) by Bobo Li, Hao Fei, Tianjie Ju, Mong-Li Lee, and Wynne Hsu. The author argues that current AI scientists are 'text prisoners': they only consume pre-digested summaries, feature vectors, and scalar scores, missing the spatial, temporal, cross-channel, and procedural relationships that drive real scientific discovery. OmniScientist addresses this with a three-layer architecture: a perception layer that ingests nine raw modalities (images, signals, audio, video, 3D structures, trajectories, tables, formulae, and graphs), plus three autonomous agents — ideation (with novelty screening, rigor checks, and execution provenance), experiment (with numerically traceable verification of claims in code), and writeup — forming a deterministic, lifecycle-wide perception pipeline. Evaluated on 36 real cases across medicine, materials science, geoscience, biology, and physics, the system completed raw-data-to-paper workflows with an average paper score of 6.3. Notably, it beat a 'blind variant' fed only precomputed features, winning 85% of head-to-head judgments across all seven evaluation dimensions, showing direct perception decisively improves discovery quality.

When the AI Scientist Opens Its Eyes: From Text Prisoner to Omni-Modal Explorer

OmniScientist: An Omni-Modal Omni-Discipline AI Scientist *Authors: Bobo Li, Hao Fei, Tianjie Ju, Mong-Li Lee, Wynne Hsu* *arXiv: 2608.13558*

Prologue: The "Scientist" Trapped in a Text Cage

Imagine being a scientist blindfolded, ears plugged, hands bound, handed a stack of printed papers and told: "Go make great discoveries."

That is the real situation of most "AI scientists" today. They can read papers, write code, run experiments, and even produce manuscripts — but they have never truly *seen* anything. They process pre-digested information: text summaries, feature vectors, scalar scores — learning about the world through third-hand hearsay.

Yet scientific discovery depends on spatial, temporal, cross-channel, and procedural relationships: the subtle morphology of cell division under a microscope, an unexpected spectral shift, subtle behavioral changes in animal video, an unanticipated protein fold, a fleeting anomalous pulse in a time-series signal. Existing AI systems are blind to exactly these factors.

Chapter 1: Perception Is Where Discovery Begins

Human science does not start with reading papers. It starts with observation: Galileo watching a swinging chandelier; Fleming noticing bacteria dying around mold on a petri dish; Penzias and Wilson hearing persistent antenna noise that turned out to be the cosmic microwave background.

Existing AI scientists sit in a library reading second-hand books. OmniScientist's core idea is simple: let the AI scientist go out and see the world with its own "eyes."

Chapter 2: Architecture — An Omni-Modal Perception System

The architecture resembles a research building with three floors:

#### Layer 1: Perception Layer

This "sensory system" consumes nine modalities as raw input (not pre-converted to text):

  • Images: microscopy, satellite imagery, medical scans
  • Signals: ECG, EEG, spectral data
  • Audio: animal calls, mechanical vibration
  • Video: behavior observation, experiment processes
  • 3D structures: molecular structures, terrain models
  • Trajectories: particle motion, pedestrian paths
  • Tables: experimental data, measurements
  • Formulae: mathematical and physical derivations
  • Graphs: networks, causal diagrams, knowledge graphs
  • Example: instead of reading "hippocampal atrophy correlates with cognitive score (r=-0.73, p<0.01)," the system sees the actual MRI scans — the specific regions of gray matter loss, white matter lesion patterns, ventricular enlargement. It perceives spatial relations and temporal evolution directly.

    #### Layer 2: Three Autonomous Agents

    Ideation Agent — generates research ideas grounded in raw perceptual evidence. Given a raw ECG waveform (not a diagnostic label), it sees subtle R-R interval variations, ST-segment elevation, T-wave asymmetry, and formulates hypotheses from those observations. It runs three checks:

  • Novelty screening against literature databases
  • Rigor checks on logical and statistical validity
  • Execution provenance: every idea traces back to specific observations
  • Experiment Agent — designs experiments, executes code, collects results. It directly runs idea, rigor, and claim checks in code: if a claim says "X correlates with Y," it loads the data itself, runs the correlation, checks p-values, and plots scatter plots. Numerical traceability ensures every number in the final paper traces to specific code lines and input data.

    Writeup Agent — assembles the full paper, shaping the narrative from the research process itself. The pipeline is deterministic but not rigid: perception influences every stage (an anomalous image region can redirect research; a contradictory data point triggers methodological review). This is "lifecycle-wide perception": observation is continuous feedback, not a one-time input.

    Chapter 3: Experiments — 36 Real-World Cases

    OmniScientist was tested on 36 real data cases spanning:

  • 5 discipline families: medicine, materials science, geoscience, biology, physics
  • 4 types of scientific evidence: images, signals, time series, structured data
  • 9 modalities
  • Key results:

  • In all 36 cases, OmniScientist completed the full raw-data-to-paper process
  • Average paper score: 6.3 (with the reference reasoning backbone) — fully automated, no human intervention
  • The authors also built a blind variant fed precomputed scalar features instead of raw data (e.g., "mean heart rate 72 bpm, QT interval 380 ms" instead of the waveform). Result:

    Direct perception beat the blind variant on all 7 evaluation dimensions, winning 85% of head-to-head judgments.

    The seven dimensions: motivation/significance, hypothesis clarity, methodological appropriateness, result support, conclusion soundness, writing quality, and overall impression.

    In other words: seeing raw data — not just summarized numbers — has a decisive impact on the quality of scientific discovery.

    Chapter 4: The Restaurant Metaphor

    A traditional AI scientist reviews a restaurant by reading reviews, the 4.5-star rating, and menu descriptions — without ever tasting the food. OmniScientist walks in, watches the chef cook, hears the room, observes other diners, tastes each dish, and only then writes its review. Which assessment would be more insightful?

    Chapter 5: A Coming of Age for AI Science

  • From symbolism to connectionism: AI learned to learn patterns from data
  • From unimodal to multimodal: AI learned to process multiple information types
  • From passive reception to active perception: AI is learning to observe the world itself
This is a philosophical shift from text-centrism to perception-centrism. Traditional AI continues a tradition from Plato to Descartes that locates truth in abstract, linguistic space. But scientific discovery tells us truth often appears first in *perception* — in an anomalous image, an unexpected signal, an unnoticed pattern. OmniScientist tries to return AI to this: perceive first, then reason; observe first, then hypothesize.

Epilogue: The Road Ahead

OmniScientist is far from perfect. The 6.3 average score shows it cannot replace human scientists, and the blind variant's non-trivial performance shows precomputed features often do capture key information. But the direction is right: future AI scientists should manipulate real lab equipment, collect field samples, and synthesize materials — a genuine agent embodied in the physical world, not just a language model on a server.

> "What I cannot create, I do not understand." — Feynman

Perhaps we should add:

> "What I cannot perceive, I cannot discover."

Perception is the starting point of discovery.

*Reference: Li, B., Fei, H., Ju, T., Lee, M.-L., & Hsu, W. (2026). OmniScientist: An Omni-Modal Omni-Discipline AI Scientist. arXiv preprint arXiv:2608.13558.*

Tags

#ai-scientist#omni-modal#perception#automated-discovery#multimodal-ai#arxiv#research-automation#paper-explainer

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633483