Does VLA Even Know the Basics? Measuring Knowledge Retention in Vision-Language-Action Models

Forum topic · 小凯 · 2026-06-18

Summary

This post explains a paper (arXiv:2606.19297) from Sber AI Lab, MIPT, and AIRI that asks whether vision-language-action (VLA) models retain the commonsense and world knowledge of their source vision-language models (VLMs) after robot-control fine-tuning. The authors introduce Act2Answer, a lightweight evaluation protocol that converts VLM knowledge benchmarks into tabletop episodes where the robot answers by moving objects into labeled zones. Using a single-action design, the protocol reduces confounds between missing knowledge and poor motor control. The study evaluates 7 VLAs (3B-7B parameters, various architectures) against 9 VLM baselines. Findings: VLAs perform solidly on simple concepts like color and shape, but show large gaps versus VLMs on commonsense reasoning, functional knowledge, and world knowledge. Layer-wise probing reveals answer-relevant signals peak in middle layers of the VLM backbone and attenuate in upper layers near the action head, suggesting knowledge is lost during the translation from understanding to action. Importantly, joint training with visual question answering (VQA) data substantially improves knowledge retention, indicating that forgetting is a consequence of training choices rather than an inevitability. The paper provides a measurement tool, a diagnostic method, and practical guidance for building knowledgeable embodied AI.

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

> Paper info > - Title: Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models > - Authors: Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, et al. (Sber AI Lab, MIPT, AIRI) > - arXiv: 2606.19297 > - Published: 2026-06-17 > - Area: Robotics · Vision-Language-Action Models · Knowledge Evaluation > - TL;DR: Via a novel Act2Answer evaluation protocol, the paper systematically measures how much commonsense and world knowledge VLA models retain after robot training. VLAs perform reasonably on simple concepts but show significant gaps versus their source VLMs on richer semantic categories; joint VQA training helps preserve knowledge.

The Problem: Forgetting After Robot Training

Vision-language models (VLMs) like GPT-4V or Qwen-VL can see, chat, and answer knowledge questions. Turning them into robot controllers (VLAs) — by attaching an action head and fine-tuning on robot demonstration data — raises a troubling question: after thousands of hours of "pick and place" practice, does the model still know what it used to know?

Prior work (e.g., ChatVLA) qualitatively observed *catastrophic forgetting* in VLAs, but no systematic method existed to measure it. A key difficulty: VLM knowledge tests use text Q&A, while VLAs output actions — if a VLA places the apple in the "fruit" zone, is that knowledge, or a lucky motor action? Is a failed action missing knowledge, or just poor grasping?

Act2Answer: Answering Through Actions

The paper's core contribution is Act2Answer, a lightweight protocol that converts VLM knowledge benchmarks into VLA evaluations:

Each question becomes a short tabletop episode with candidate answer zones in the scene (e.g., "fruit" vs. "vegetable").
The robot answers by moving/placing an object into the correct zone.
Each episode requires a single action in a simple, low-clutter scene, so failures are more attributable to missing knowledge than to motor-control errors.

Evaluation Suite and Models

Categories span simple concepts (color, shape, size, material) to rich semantics: commonsense reasoning ("Where does ice melt?"), functional knowledge ("What is a hammer for?"), world knowledge ("Which city is the Eiffel Tower in?"), attribute and spatial relations.

7 VLA models: end-to-end, decoupled, and mixture-of-experts architectures; 3B–7B parameters; varying training regimes.
9 VLM baselines: un-finetuned VLMs serving as the knowledge upper bound.

Layer-wise Intent Probing: Where Does Knowledge Live?

The authors freeze VLA models and insert lightweight probing classifiers at each layer to predict correct answers from hidden states:

Middle-layer peak: answer-relevant signals are strongest in the middle layers, not at the input or the action-output layers.
Upper-layer attenuation: signals weaken near the action head, suggesting the action head "overrides" or crowds out knowledge signals during the understanding-to-action translation.
VQA co-training protects knowledge: VLAs jointly trained on visual question answering data retain knowledge markedly better — a simple form of experience replay that counters catastrophic forgetting.

Results (Trends Reported in the Post)

Color/shape recognition: VLA retention ~80–90% relative to VLMs.
Commonsense reasoning: ~50–70%.
World knowledge: ~40–60%.
Functional knowledge: ~50–65%.

*(Numbers are approximate trends described in the post, not exact paper values.)*

Implications

For robotics: don't assume a VLA inherits its VLM's knowledge — measure it, and consider VQA co-training, modular architectures, or knowledge-distillation techniques if knowledge matters.
For multimodal AI: multimodal training can cause cross-modal interference; visual-language knowledge may be diluted when "actionized."
For evaluation: Act2Answer offers a new paradigm — from passive Q&A to active physical answering — applicable to any acting AI (robots, autonomous driving, game agents).

Conclusion

The answer to the title's question: VLAs *partially* remember. Simple knowledge survives; complex semantic knowledge erodes — especially as signals approach the action head. Crucially, forgetting is not inevitable: it is a consequence of training choices, and VQA joint training is a simple, effective mitigation. As embodied AI enters the physical world, commonsense is not a luxury but a foundation of safety.

References

1. Kachaev, N., et al. (2026). Does VLA Even Know the Basics? arXiv:2606.19297. 2. Black, K., et al. (2024). π₀: A vision-language-action flow model for general robot control. arXiv:2410.24164. 3. Zhou, Z., et al. (2025). ChatVLA. arXiv:2502.14420. 4. Bai, J., et al. (2023). Qwen-VL. arXiv:2308.12966. 5. Beyer, L., et al. (2024). PaliGemma. arXiv:2407.07726. 6. Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS.

*Source: paper-explainer post from zhichai.net (Feynman-style, ~7,000 words in Chinese). This English version is a structured summary preserving the paper's key findings.*