IVLR: Interleaved Vision-Language Reasoning for Long-Horizon Robot Manipulation

Paper: Thinking in Text and Images: Interleaved Vision-Language Reasoning Traces for Long-Horizon Robot Manipulation Authors: Jinkun Liu, Haohan Chi, Lingfeng Zhang, Yifan Xie, YuAn Wang arXiv: 2605.00438 | 2026-04-29

1. The Robot That Acts Without Thinking

Imagine a robot making breakfast. Most existing VLA (Vision-Language-Action) policies work like this:

• Observe the kitchen scene
• Directly output an action: "move gripper above the egg"
• No explicit plan, no explanation of "why"

Problems arise immediately:

• If the egg is behind the fridge, does the robot know?
• If the pan isn't hot yet, does it wait?
• If a step goes wrong, can it backtrack?

Existing approaches either hide the plan inside a latent state or expose only a single modality.

2. Two Core Challenges of Long-Horizon Manipulation

1. Logical consistency vs. geometric grounding

• Text reasoning knows "crack the egg before frying it" — but not where the egg or pan is. It lacks spatial constraints.

2. Visual prediction vs. semantic constraints

• Visual prediction knows where objects are — but not why a particular object should be grasped. It lacks causal reasoning.

The blind spot of existing methods:

• Text CoT: logic without space
• Visual prediction: space without logic
• The two remain separate and cannot cooperate

3. IVLR: Interleaved Vision-Language Reasoning



Core idea: let the robot alternate between textual reasoning and visual reasoning, forming an "interleaved reasoning chain."

Technical approach:

1. Text reasoning steps

• "I need to make a fried egg"
• "Step 1: get the egg" / "Step 2: crack it into a bowl"
• Provides causal ordering and logical constraints

2. Visual reasoning steps

• [Image: inside the fridge] — "the egg is on the right of the second shelf"
• [Image: the stove] — "the pan is on the stove, not yet heated"
• Provides geometric grounding and spatial information

3. Interleaved execution

• Text: "get the egg" → Vision: localize the egg
• Text: "crack the egg" → Vision: confirm the bowl's position
• Alternating, mutually complementary steps

4. Policy framework

• Actions are generated from the interleaved reasoning trace
• Each reasoning step guides the next action
• A complete "think–look–act" loop

This mirrors how humans handle complex tasks: think about what to do, look for where things are, act, then think about the next step — and repeat.

4. Why Interleaving Beats a Single Modality

Pure text CoT fails at execution:

• "Grab the egg" — but where is it?
• "Turn on the stove" — but where is it?
• Good plans, poor grounding.

Pure visual prediction lacks purpose:

• It knows "the gripper should move left" but not why
• It cannot handle unexpected situations

IVLR's advantages:

• Bidirectional constraints: text plans guide visual attention ("I need the egg" → the visual system focuses on eggs); visual feedback revises text plans ("the egg isn't where expected" → update the plan)
• Explainability: every step has a textual justification — "why did I do this action?" — aiding debugging and improvement
• Error recovery: on failure, the robot can backtrack the reasoning chain: "Step 3 failed, let me return to Step 2 and re-evaluate"

5. A Feynman-Style Judgment: Thinking Requires a Multimodal Loop

Feynman famously said:

> "What I cannot create, I do not understand."

Applied to manipulation: a robot cannot think in text alone, nor react with vision alone. Humans constantly switch between "thinking" and "looking" while acting. IVLR imitates this natural cognitive loop — text provides logic, vision provides grounding, and the two interleave into complete understanding.

This echoes a basic insight from cognitive science: human cognition is multimodal; language and vision are not separate systems but work together and reinforce each other.

6. Takeaways

If you are building robots or decision systems, ask:

1. Does my system use both logical reasoning and spatial perception? 2. Is the reasoning process interleaved and dynamic? 3. Can the system explain *why* it takes an action? 4. When errors occur, can it backtrack and revise its plan?

IVLR reminds us that intelligence is not just "doing the right thing" but "knowing why." When a robot learns to think alternately in text and images, it transforms from a reactive machine into a reflective agent. For the complex world of long-horizon manipulation, this "think while doing" ability is indispensable.

In the future of robotics, the best operator is not the fastest one — it's the one that thinks best.