OmniAct: From Isolated Skills to Everyday Physical Autonomy

Paper: Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy Authors: Junhao Shi, Zezheng Huai, Siyin Wang, Jia Chen, Yubang Wang arXiv: 2606.27251 | Areas: cs.RO, cs.AI

Key points

• The problem: A single everyday request like "make breakfast, set the AC to 26 degrees, check the weather" spans the physical world (robot manipulation), the digital/IoT world (smart-home APIs), and the web (online queries). Most current systems handle only one of these domains in isolation.
• Three weaknesses of existing systems:
• VLM planners lack grounding: they see the global scene but miss execution details (is the cup empty? is the dishwasher door open? is the path blocked?).
• Unbounded context growth: all history is crammed into one context window, degrading performance as tasks grow longer.
• Open-loop VLA execution: actions are executed as planned without checking whether they actually achieve the intended result.

The OmniAct architecture

OmniAct is not a bigger brain but a better organization — three cooperating modules:

1. Multimodal Semantic Planner

Acts like a conductor: it understands the user's goal and decomposes it into subtasks assigned to heterogeneous actuators (robot arm, APIs, IoT devices). The key innovation is a unified action space: every action — physical or digital — is described as "in what state, on what object, perform what operation."

2. Adaptive Hierarchical Memory

A three-tier structure mirroring human memory:
• Working memory — currently active information
• Episodic memory — event-organized history
• Semantic memory — long-term common sense and preferences

The standout mechanism is event-boundary-driven compression: rather than truncating at fixed intervals, the system detects event boundaries (e.g., "making breakfast" → "answering the door") and semantically compresses completed events into summaries, keeping key facts and discarding redundancy. A semantic importance assessment distinguishes one-off instructions from lasting preferences (e.g., a coffee allergy), deciding what to keep indefinitely. Result: token consumption stays nearly flat over 100,000+ cumulative tokens — near O(1) memory growth instead of O(n).

3. Asynchronous Visual Preemption Engine

While actions execute, this module asynchronously observes the environment and preempts and replans when reality diverges from the plan. It upgrades classic closed-loop control to semantic loop closing: checking not "did the arm reach the position?" but "was the cup actually placed in the dishwasher?" — verifying outcomes, not motions.

Experimental results

• Setup: 40 real-world long-horizon tasks across two robot platforms and four IoT devices, at three complexity tiers (single-skill, multi-skill composition, long-horizon with failure/exception handling).
• Headline finding: a mid-scale open-source model running under OmniAct matches proprietary closed-source models in end-to-end success rate.
• Stable gains across all complexity levels, flat token consumption, and seamless cross-platform transfer (robot arms, mobile robots, IoT).

The takeaway: architecture can matter more than scale — a well-organized small team can outperform a chaotic large one.

Why it matters

• A shift from a monolithic "super brain" to distributed intelligence with clear interfaces — analogous to how modern cities outperform rule-by-omnipotent-monarch.
• Emphasis on semantic closed-loop control as a missing ingredient in current VLA systems.
• The layered memory design echoes human cognitive-science models, suggesting that *how information is organized* may matter more than *how much* is stored.

OmniAct is a framework rather than a product, but it sketches a pragmatic path toward general AI assistants: not one omnipotent oracle, but a well-organized team — a planner, an archivist, and a quality inspector that know when to cooperate, when to divide work, and when to stop and rethink.

Reference

> Shi, J., Huai, Z., Wang, S., Chen, J., & Wang, Y. (2026). Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy. *arXiv preprint arXiv:2606.27251*.