Does a Robot's VLA Model Still Remember Commonsense? Measuring Knowledge Retention After Action Training

Forum topic · 小凯 · 2026-06-18

Summary

A forum post discusses a paper introducing Act2Answer, a lightweight evaluation protocol that converts vision-language model (VLM) knowledge benchmarks into robot-action-based tests for Vision-Language-Action (VLA) models. Since VLAs output actions rather than text, standard QA evaluation cannot measure whether they retain commonsense and world knowledge after robotics fine-tuning. Act2Answer reformulates each question as a single tabletop placement action, where choosing a spatial region corresponds to selecting an answer, reducing confounds from motor-control errors. The study evaluates 7 VLA models against 9 source VLMs. Findings: VLAs perform reasonably on simple concepts like color and shape, but show large gaps on complex semantic categories such as commonsense reasoning, functional knowledge, and world knowledge. Layer-wise probing reveals that answer-related signals peak in middle layers of the VLM backbone and attenuate in upper layers near the action head, suggesting knowledge is overwritten during action translation. Joint training with VQA data significantly improves knowledge retention, indicating forgetting is a consequence of training choices rather than inevitability. The post also situates the work within catastrophic forgetting research and discusses implications for robot safety, multimodal AI, and evaluation methodology.

When a Robot's 'Brain' Gets Reprogrammed: Does a VLA Model Still Remember Commonsense?

*English translation and structured summary of a Chinese forum post discussing: "Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models" (Nikita Kachaev et al., Sber AI Lab, MIPT, AIRI; arXiv:2606.19297).*

Key points

The problem: When strong vision-language models (VLMs) are fine-tuned into robot controllers (Vision-Language-Action models, VLAs), how much of their original commonsense and world knowledge survives? Prior work (e.g., ChatVLA) noted catastrophic forgetting qualitatively, but no systematic measurement existed.
Why it's hard: VLAs output actions, not text. If a robot places an apple in the "fruit" bin, is that knowledge or a lucky random motion? A failed action may reflect poor control rather than missing knowledge.

Act2Answer: answering through actions

Each knowledge question becomes a short tabletop episode with candidate answer regions; the robot answers by placing an object in the correct spatial region.
Single-action episodes on simple tabletop scenes minimize motor-control confounds, so failures more plausibly indicate missing knowledge.

Evaluation setup

Knowledge categories tested: simple concepts (color, shape, basic attributes) and complex semantic categories (commonsense reasoning like "ice melts in the sun," functional knowledge, world knowledge, attribute and spatial relations).
7 VLA models (3B–7B parameters; end-to-end, decoupled, and mixture-of-experts architectures) compared against 9 unmodified VLMs as the knowledge upper bound.

Findings

Simple vs. complex: VLAs retain solid performance on color/shape, but show large gaps vs. source VLMs on commonsense and world knowledge.
Layer-wise probing: lightweight probes inserted at each layer show answer-related signals peak in middle layers of the VLM backbone and attenuate in upper layers near the action head — knowledge appears to be overwritten during translation from understanding to action.
VQA joint training helps: VLAs co-trained on visual-question-answering data retain significantly more knowledge, acting as a simple replay-style defense against catastrophic forgetting (no extra mechanisms needed).

Discussion highlights

VLA training resembles lossy knowledge distillation: task-relevant skills survive, general knowledge erodes. Whether retention matters is task-dependent — but for open-world, safety-critical robots (fragile objects, hot soup, medicine around children), commonsense is a safety foundation, not a luxury.
Act2Answer suggests a broader evaluation paradigm shift: from passive QA to action-based assessment, applicable to any embodied AI (robots, autonomous driving, game agents).
The takeaway is optimistic: forgetting is a consequence of training strategy, not inevitability — VQA co-training, modular architectures, and replay methods can preserve knowledge while robots learn to act.

References cited in the post

Kachaev et al., *Does VLA Even Know the Basics?* arXiv:2606.19297
Black et al., *π₀: A vision-language-action flow model for general robot control*, arXiv:2410.24164
Zhou et al., *ChatVLA*, arXiv:2502.14420
Bai et al., *Qwen-VL*, arXiv:2308.12966
Beyer et al., *PaliGemma*, arXiv:2407.07726
Kirkpatrick et al., *Overcoming catastrophic forgetting in neural networks* (EWC), PNAS 2017