Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Models

Forum topic · 小凯 · 2026-09-09

Summary

This arXiv paper (2609.05369) by Vivek Chavan, Yahuan Shi, Oliver Heimann, Kevin Haninger, and Jörg Krüger addresses the brittleness of vision-language-action (VLA) models in long-horizon robotic manipulation. The authors propose a neuro-symbolic framework combining learned VLA control with explicit task graphs and multimodal procedural memory. Task graphs encode action dependencies, valid transitions, and branch conditions, while memory tracks the active step, completed actions, textual context, and task-relevant visual evidence. These structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions. Human demonstrations supply additional spatial-temporal guidance via gaze or saliency cues; to isolate effects on policy learning, the initial study bypasses cross-view gaze transfer and annotates pseudo-gaze directly on robot-view teleoperation videos for VLA fine-tuning and inference. Evaluation covers workspace cleanup and surgical instrument handling, measuring object/destination selection, subtask completion, task progress, step-order consistency, full-task success, and procedural or execution errors, positioning symbolic reasoning plus demonstration-derived visual guidance as complementary mechanisms for reliable long-horizon VLA operation.

Paper Overview

Field: Computer Vision (CV) Authors: Vivek Chavan, Yahuan Shi, Oliver Heimann, Kevin Haninger, Jörg Krüger Published: 2026-09-04 arXiv: 2609.05369

Abstract

Vision-language-action (VLA) models can execute short manipulation skills, but remain brittle in long-horizon procedures requiring persistent task state, dependency-aware reasoning, conditional decisions, and reliable grounding. The authors investigate a neuro-symbolic framework that combines learned VLA control with explicit task graphs and multimodal procedural memory.

Task graphs encode action dependencies, valid transitions, and branch conditions.
Multimodal procedural memory maintains the active step, completed actions, textual context, and task-relevant visual evidence.
Together, these structures guide object selection, destination grounding, subgoal dispatch, and verification of expected state transitions.
Human demonstrations provide additional spatial and temporal guidance through gaze or saliency cues.

To isolate the effect of this guidance on policy learning, the initial study bypasses cross-view gaze transfer and instead annotates pseudo-gaze directly on robot-view teleoperation videos. This supervision is then used for VLA fine-tuning and inference.

Evaluation Domains

The framework is studied on two long-horizon manipulation domains:

1. Workspace cleanup 2. Surgical instrument handling

Both require ordered execution, visual grounding decisions, and conditional branching. Evaluation metrics include correct object and destination selection, subtask completion, task progress, step-order consistency, full-task success, and procedural or execution errors.

Conclusion

This work positions structured symbolic reasoning and demonstration-derived visual guidance as complementary mechanisms for reliable long-horizon VLA operation.

---

*Auto-collected on 2026-09-09.*