OmniAct: A Framework Enabling AI Agents to Move Freely Between Digital and Physical Worlds

Forum topic · 小凯 · 2026-06-28

Summary

OmniAct, presented in the paper 'Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy' (arXiv:2606.27251), is a framework that unifies physical robot manipulation, IoT device control, and web/API tasks within a single embodied agent architecture. It addresses three key weaknesses of existing systems: VLM planners that lack fine-grained situational awareness, unboundedly growing context windows, and VLA policies that execute actions without verification. OmniAct introduces three cooperating components: a multimodal semantic planner with a unified action space spanning physical and digital actions, an adaptive hierarchical memory with working/episodic/semantic layers that compresses information at event boundaries, and an asynchronous visual preemption engine performing semantic loop closing to verify whether task goals are actually achieved. Experiments across 40 real-world long-horizon tasks on two robot platforms and four IoT devices show that mid-sized open-source models equipped with OmniAct match proprietary closed-source models in end-to-end success rate, while token consumption remains nearly flat beyond 100,000 cumulative tokens—approaching O(1) memory complexity. The results suggest that architectural innovation and organized multi-module intelligence can matter more than raw model scale on the path toward general embodied autonomy.

OmniAct: When Robots Learn to Be in Multiple Worlds at Once

> Paper: Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy > Authors: Junhao Shi, Zezheng Huai, Siyin Wang, Jia Chen, Yubang Wang > arXiv: 2606.27251 > Areas: cs.RO, cs.AI

---

1. Opening: A Not-So-Fanciful Sci-Fi Scenario

Imagine waking up and telling your home assistant: "Prepare breakfast, set the living room AC to 26 degrees, and check if it will rain today."

Sounds simple—but for AI, this is a nightmare-level task:

Preparing breakfast requires physical manipulation: opening the fridge, taking out ingredients, operating the stove.
Adjusting the AC requires IoT control: sending commands to smart-home APIs.
Checking the weather requires a web query: calling an external service.

These belong to the physical, digital, and online worlds respectively. Most existing AI systems handle only one of these—they are isolated tribes on separate islands, speaking different languages. OmniAct aims to unify them into a single framework that moves seamlessly across all three worlds.

2. Why Is This Hard?

2.1 The "Omnicompetence" Trap

Knowing *of* things and *doing* them well are separated by a deep gulf. A VLM may know what a coffee cup and a dishwasher are, yet lack:

Action sequencing: planning walk → grasp → turn → place
Failure recovery: what if a chair blocks the path?
Memory management: remembering what it saw and did
Physical common sense: glass cups can't be squeezed hard; dishwasher doors must be opened first

2.2 Three Disabilities of Existing Systems

1. VLM planners see the big picture but miss the details—like directing ground workers from 10,000 meters up. Is the cup empty? Is the dishwasher door open? Is the arm too short without moving a chair? 2. Unboundedly growing context—all history is stuffed into one ever-fuller context window until it becomes an intractable tangle. 3. VLA policies walk with eyes closed—they pre-plan actions and execute blindly, never checking whether reality diverges from the plan.

3. OmniAct: A Three-Story Palace

Rather than a smarter single brain, OmniAct builds a smarter organization.

3.1 Multimodal Semantic Planner — the orchestra conductor

The planner doesn't play instruments itself; it reads the score (understands your intent) and distributes parts ("robot arm handles the eggs, API handles the weather, IoT handles the AC").

Key innovation: unified action space. Physical actions (grasping, moving) and digital actions (API calls, database queries) are expressed in one common language: *in what state, on what object, perform what operation*.

3.2 Adaptive Hierarchical Memory — a self-organizing room

Three tiers:

Working memory: the desk—what's currently being handled
Episodic memory: the filing cabinet—organized by event ("last week's grocery run")
Semantic memory: common-sense knowledge ("cups are for drinking")

Key innovation: event-boundary-driven compression. Instead of mechanically truncating at fixed intervals, OmniAct compresses like humans do—at event boundaries. Finishing breakfast and answering the doorbell are two events; breakfast details get summarized into episodic memory while door-related details stay in working memory. This is *semantic* compression: keep the key facts, discard the redundancy.

Mathematical elegance. Over 100,000+ cumulative tokens of interaction, OmniAct's token consumption stays nearly flat. Where existing systems grow at O(n), OmniAct approaches O(1)—like a drawer that automatically compresses old notes into summaries, never filling up.

Memory freshness and expiry. Via semantic importance assessment, one-off instructions ("put the remote on the table") expire; strategic knowledge ("the user is allergic to coffee") persists—just as the brain distinguishes "what to eat today" from "my own name."

3.3 Asynchronous Visual Preemption Engine — the good navigation system

While the robot acts, this engine asynchronously observes the environment. If reality deviates, it preempts the current action and replans.

Key innovation: semantic loop closing. Classical closed-loop control checks "did the arm reach the target position?" Semantic loop closing checks "did the cup actually end up in the dishwasher?"—verifying outcomes, not motions.

4. Experiments: Small Models Beat Big Ones

Setup

40 real-world long-horizon tasks across two robot platforms and four IoT devices, at three difficulty levels:

🟢 Simple: single skill, short flow ("put the cup on the table")
🟡 Medium: skill composition ("make breakfast and clean the kitchen")
🔴 Complex: long-horizon planning with failure handling ("prepare dinner, check all doors/windows, raise an alarm on gas leak")

Core results

A mid-sized open-source model with OmniAct matches proprietary closed-source models in performance.
End-to-end success rates improve steadily across all difficulty levels.
Token consumption stays nearly flat over 100,000+ cumulative tokens.
Cross-platform transfer: the same framework moves seamlessly between robot arms, mobile robots, and IoT devices.

Why it matters

The prevailing assumption is *bigger is better*. OmniAct's results show architectural innovation can outweigh model scale—a well-organized small team can beat a chaotic giant corporation.

5. Deeper Reflections

From centralized to distributed intelligence: abandoning the fantasy of one super-brain in favor of specialized modules with clean interfaces—like modern cities run by departments, not an all-powerful monarch.
From open-loop to closed-loop—not just physically but semantically: constantly asking "is what I'm doing aligned with the goal?"
Hierarchical memory mirrors human cognition (working / episodic / semantic), suggesting that the essence of intelligence may lie in how information is organized, not how much there is.

6. Closing Thoughts

OmniAct is a framework, not a product, but it sketches the future AI assistant:

Not an omniscient oracle, but a well-organized team—a conductor (planner), an archivist (memory), and an inspector (verification engine).

This may be a more pragmatic path toward AGI: not building an omnipotent *brain*, but an omnipotent *organization*. After all, civilization's great achievements come not from lone geniuses but from millions of ordinary people organized effectively. AI may need to walk the same road.

Addendum: When Robots Acquire Professionalism

A great employee isn't great because they remember every detail of the past decade, but because they know what to remember and what to forget: keep client preferences (semantic memory), let go of last week's meeting details (episodic memory), track current project status (working memory).

OmniAct's three-tier memory is essentially teaching robots "professionalism"—remembering *better*, not *more*. Perhaps future AI assistants will be judged not by how many tokens they can process, but by how much they can forget.

After all, human wisdom lies largely in knowing when to let go.

---

Reference

> Shi, J., Huai, Z., Wang, S., Chen, J., & Wang, Y. (2026). Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy. *arXiv preprint arXiv:2606.27251*.