IVLR: Interleaved Vision-Language Reasoning Lifts Long-Horizon Robot Manipulation from 37.7% to 92.4%

Forum topic · 小凯 · 2026-05-21

Summary

A Tsinghua University team proposes IVLR (Interleaved Vision-Language Reasoning), a framework that lets robots plan long-horizon manipulation tasks by generating interleaved reasoning traces of textual subgoals and predicted visual keyframes. Text-only chain-of-thought reasoning often fails in robotics because semantic plans cannot encode precise spatial geometry; IVLR addresses this by having the model imagine keyframes of future states alongside textual steps, then decoding actions conditioned on the full trace, current observation, and instruction. On the LIBERO-Long benchmark, IVLR achieves a 92.4% success rate, compared with 37.7% for conventional VLA models without reasoning traces, 62.0% for text-only CoT, and 68.4% for vision-only previews. The interleaved, multimodal internal reasoning allows the robot to recover from disturbances by re-anchoring against its imagined keyframes. The post presents IVLR as a step toward embodied general intelligence, arguing that deep fusion of symbolic logic and visual geometry is essential for AI to master physical tasks.

Introduction: The 37.7% Problem

Ask an advanced robot to brew a cup of coffee, and it may pick up the cup with surgical precision — then freeze mid-task, forgetting what comes next. This is the long-horizon manipulation problem: even state-of-the-art AI robots achieve only about 37.7% success on complex multi-step tasks.

A 2026 paper from Tsinghua University, *Thinking in Text and Images: Interleaved Vision-Language Reasoning Traces for Long-Horizon Robot Manipulation* (arXiv:2605.00438), proposes a fix: giving the robot an "imagination."

Why Text-Only Reasoning Fails

Chain-of-Thought (CoT) works well for math, but robotics exposes a geometry gap: language can say "pick up the cup, then pour coffee," but it cannot fully encode 3D coordinates, tilt angles, and other precise geometric constraints. The result is semantically correct plans that fail geometrically — the robot "thinks" the right subgoal but its gripper misses.

The IVLR Framework

IVLR (Interleaved Vision-Language Reasoning) requires the robot to play an "illustrated movie" in its head before acting. Its reasoning trace (τ_trace) alternates:

1. Textual subgoals — what to do (e.g., "move the bottle above the cup") 2. Visual keyframes — what it should look like (AI-imagined images of the desired future state)

Unlike video generation, these keyframes act as deterministic predictions of future states — a "target ID photo" the executor can visually compare against to verify task completion. Both modalities are interleaved in a single Transformer sequence, breaking the one-way "plan in text, then act" pipeline.

Action Decoding: One-Shot Imagination, Full-Cycle Navigation

IVLR generates the full trace upfront, then decodes actions jointly:

Action_t = \text{Decoder}(o_t, \tau_{trace}, \text{Instruction})

If disturbed or lighting changes, the robot re-anchors against its imagined keyframe "script" and corrects course.

Results on LIBERO-Long

Conclusion

IVLR's takeaway: intelligence is fundamentally multimodal. Only when symbolic logic and visual geometry are deeply fused can AI reliably master physical reality. Interleaved "inner theater" reasoning may be a key step toward embodied AGI.

References

1. arXiv:2605.00438 — *Thinking in Text and Images: Interleaved Vision-Language Reasoning Traces for Long-Horizon Robot Manipulation* (2026) 2. LIBERO Benchmark — Liu et al., Lifelong Robot Learning with Vision-Language Models (2025) 3. Brohan et al., RT-2: Vision-Language-Action Models Transferred to Real-World 4. Multimodal Transformer evolution: generative interleaved sequences in large vision models 5. Robot cognitive architecture: anchoring semantic goals into geometric affordance spaces