Smart Brain, Clumsy Hands: VLM Capability Poorly Predicts Robot Control Performance

Forum topic · 小凯 · 2026-10-24

Summary

A study from Tsinghua University's Institute for Interdisciplinary Information Sciences and the Alibaba Qwen team (VLM4VLA, arXiv 2601.03309) finds that the general capabilities of vision-language models (VLMs) are surprisingly poor predictors of downstream robot control performance. The researchers attached nine open-source VLMs—including Qwen2.5-VL, Qwen3-VL, Paligemma, and Kosmos-2—to an identical robot policy backbone with minimal added parameters, then evaluated them in three simulation benchmarks: Calvin, SimplerEnv, and Libero. Rankings differed across all three environments: the smallest model, 1.7B-parameter Kosmos-2, topped SimplerEnv (60.4) and Libero-10 (55.0), while the Qwen-VL family led on Calvin. Auxiliary embodied fine-tuning tasks (embodied QA, pointing, depth estimation) failed to improve control, even degrading Calvin scores. Ablations locate the bottleneck in the visual encoder rather than the language side: injecting control-relevant supervision into the visual encoder via FAST action tokenization raised SimplerEnv-Bridge scores from 27.6 to 45.7. VLM pretraining remains necessary—training from scratch performs worst—but VQA leaderboard rankings should not be used to select backbones for vision-language-action models.

Smart Brain ≠ Nimble Hands

A common default assumption in embodied AI over the past two years: give a robot the strongest vision-language model (VLM), and it will do better work. Tsinghua University's Institute for Interdisciplinary Information Sciences and the Alibaba Qwen team dismantled that assumption experimentally in their paper (VLM4VLA, arXiv 2601.03309). Their finding: a VLM's intelligence has unexpectedly weak predictive power for downstream robot control performance.

Experimental Design

The setup is clean. The researchers built a minimal adaptation pipeline: they connected nine open-source VLMs—Qwen2.5-VL (all variants), Qwen3-VL (multiple sizes), two generations of Paligemma, and Kosmos-2—to the same robot policy backbone, adding only a small set of new parameters while freezing the rest, so that the comparison isolates the "brain" itself rather than other components. The exam was then run in three simulation environments: Calvin, SimplerEnv, and Libero.

Results Defy Intuition

On SimplerEnv's tabletop pick-and-place and Libero's long-horizon tasks, the top scorer was not any large model, but the smallest one: Kosmos-2, at just 1.7B parameters—four times smaller than Qwen2.5-VL-7B. It ranked first on both leaderboards: 60.4 average on SimplerEnv and 55.0 on Libero-10, beating the entire Qwen family and Paligemma. Yet on Calvin's sequential tasks, the ranking completely flipped, with the Qwen-VL series significantly ahead.

Three environments produced three different rankings. Correlation between general-capability benchmarks and robot performance existed only on Calvin; on the other two environments, there was no correlation at all—despite high VQA benchmark scores being the primary criterion researchers use when selecting a "brain."

Embodied Fine-Tuning Doesn't Help

A second round of experiments was even more sobering. Since general capability doesn't predict control, would tutoring the brain toward embodiment help? The team selected seven auxiliary embodied tasks—embodied question answering, visual pointing, depth estimation, and similar—and fine-tuned the VLMs on each before reinstalling them. The result: all seven subjects were a waste. Post-tutoring models performed *worse* on Calvin, with drops ranging from 0.009 to 0.096. None of the exam-crammed material showed up on the test.

The Bottleneck Is the Visual Encoder

Ablation studies deliver the paper's most valuable insight: what happens on the language side doesn't matter—whether word embeddings are frozen or fine-tuned makes almost no difference. The real bottleneck is the visual encoder. When the researchers injected control-relevant supervision directly into the visual encoder (training this module with the FAST action tokenizer) before downstream training, performance rose consistently—even when the encoder was fully frozen during downstream training. When it was unfrozen and trained jointly, SimplerEnv-Bridge scores jumped from 27.6 to 45.7—an eighteen-point gain.

Translation: the vision module of a VLM learns "describe what you see" during pretraining, while robots need "act on what you see." These are fundamentally different kinds of vision, and the gap is large enough to warrant its own dedicated training.

This line of reasoning also explains a known plugin phenomenon: pi0, built on a Paligemma-1 backbone, performed roughly the same as the bare backbone when not fed state information—no matter how refined the action-expert add-on, its ceiling was capped by the backbone's visual representations.

Takeaway

The paper appeared on arXiv on January 6 and was updated in late May—nine months ago, though it was recently presented as "new." Its conclusions have not aged; if anything, they have hardened with time. VLM pretraining is a necessary condition—control groups trained from scratch ranked last across the board—but treating VQA leaderboard rankings as the basis for choosing a robot's brain will measure something else entirely. Intelligence answers "can you articulate it?"; control asks "where does the hand go?" They share a pair of eyes, but not the same pair of glasses.