Do VLA Models Still Remember Commonsense? Measuring Knowledge Retention After Robot Training

Forum topic · 小凯 · 2026-06-18

Summary

This post analyzes the paper 'Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models' (arXiv:2606.19297), which systematically measures how much commonsense and world knowledge vision-language-action (VLA) models retain after being fine-tuned into robot controllers. The authors from Sber AI Lab, MIPT, and AIRI propose Act2Answer, a lightweight evaluation protocol that converts VLM knowledge benchmarks into tabletop action episodes, where the robot 'answers' multiple-choice questions by placing objects into labeled zones. This reduces the confound between knowledge gaps and motor control failures. Evaluating 7 VLA models against 9 source VLMs, the study finds solid retention on simple concepts like colors and shapes but larger gaps in semantic categories such as commonsense reasoning, functional knowledge, and world knowledge. Layer-wise probing reveals that answer-related signals peak in middle layers and attenuate in upper layers near the action head, suggesting knowledge is lost during the translation from understanding to action. Importantly, VLA models jointly trained with visual question answering (VQA) data retain significantly more knowledge, indicating that forgetting is a consequence of training choices rather than an inevitability. The post includes a full discussion of evaluation design, catastrophic forgetting, and implications for robotics.

Overview

This forum post presents a detailed walkthrough of the paper "Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models" (arXiv:2606.19297), by Nikita Kachaev and colleagues from Sber AI Lab, MIPT, and AIRI.

The central question: when a powerful vision-language model (VLM) is fine-tuned into a vision-language-action (VLA) robot controller, does it retain the knowledge it originally had—or has it been reprogrammed into pure motor instinct?

The Problem

Existing work (e.g., ChatVLA) has qualitatively observed catastrophic forgetting in VLA training, but there was no systematic way to *measure* it. Evaluating VLA knowledge is fundamentally hard: VLAs output actions, not text. If a robot places an apple in the "fruit" zone, did it do so because it knows apples are fruit, or by random chance? Conversely, a failed action may reflect poor motor control rather than missing knowledge.

Act2Answer: Answering Through Action

The paper introduces Act2Answer, a lightweight protocol that converts VLM knowledge benchmarks into tabletop action episodes:

Each question becomes a short episode with candidate answers represented as spatial zones or objects on a tabletop
The robot answers by executing a single placement/reach action toward the correct answer region
Success rates measure knowledge retention

Because each episode requires only one simple action in a low-clutter scene, failures are more likely attributable to missing knowledge than to control errors.

Evaluation Setup

7 VLA models spanning different architectures (end-to-end, decoupled, mixture-of-experts), scales (3B–7B), and training regimes (robot-only data vs. VQA joint training)
9 VLM baselines as the knowledge upper bound
Knowledge categories: simple concepts (color, shape, size, material) and richer semantic categories (commonsense reasoning, functional knowledge, world knowledge, attribute relations, spatial relations)

Key Findings

Simple concepts survive: VLAs show solid retention of colors and shapes, approaching VLM baselines (roughly 80–90% relative retention per the post's trend figures)
Semantic knowledge degrades: commonsense reasoning (~50–70%), world knowledge (~40–60%), and functional knowledge (~50–65%) show larger gaps vs. source VLMs
Layer-wise probing: answer-relevant signals peak in middle layers of the VLA and attenuate in upper layers near the action head—suggesting the action head overrides or dilutes knowledge signals during the understanding-to-action translation
VQA joint training helps: VLAs co-trained on visual question answering data retain significantly more knowledge, acting as a simple replay-style defense against catastrophic forgetting

Discussion and Implications

The post frames VLA training as a form of *lossy* knowledge distillation: only task-relevant knowledge survives. It discusses the trade-off—open-world robots arguably need commonsense for safe behavior (e.g., knowing ice melts in the sun, hot soup damages plastic), while others argue operational tasks need skills more than encyclopedic knowledge. The paper takes no side, instead providing:

1. Act2Answer — a measurement tool for knowledge loss 2. Layer-wise probing — a diagnostic for *where* knowledge disappears 3. Empirical evidence — VQA joint training as a simple, effective mitigation

The takeaway: forgetting in VLAs is not inevitable—it is a consequence of training choices. Knowledge retention should be measured, not assumed.

References (as cited in the post)

Kachaev, N., et al. (2026). Does VLA Even Know the Basics? arXiv:2606.19297
Black, K., et al. (2024). π₀: A vision-language-action flow model. arXiv:2410.24164
Zhou, Z., et al. (2025). ChatVLA. arXiv:2502.14420
Bai, J., et al. (2023). Qwen-VL. arXiv:2308.12966
Beyer, L., et al. (2024). PaliGemma. arXiv:2407.07726
Kirkpatrick, J., et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS.

*Note: The post is a Feynman-style tutorial interpretation with life-like analogies; numerical retention figures above are approximate trends as described by the author, not exact paper values.*