Key points
Part 1: VLA vs. VLM
- Fundamental difference: VLMs "see and speak" (output is always text); VLAs "see and act" (output motor/joint commands). Core line: VLM outputs text, VLA outputs actions.
- Architectures:
- VLMs: pre-trained LLM backbone + ViT visual encoder + projection layers. Lineage: CLIP (2021, contrastive alignment) → Flamingo/BLIP (generative) → LLaVA (2023, now the standard recipe) → Qwen2-VL, LLaMA 3.2 Vision. Newer work like Emu3 explores "native multimodality" with unified token processing.
- VLAs modify the output layer to generate actions. Main paradigms:
- End-to-end (RT-1, RT-2, OpenVLA): image → motor commands directly; simple but limited generalization.
- Dual-system (NVIDIA GR00T N1): fast reflex "System 1" (~10ms latency) + slow reasoning "System 2".
- Hierarchical (CogACT, NaVILA): LLM for planning, dedicated controllers for execution; modular but latency accumulates.
- Self-correcting (SC-VLA): fast path by default, LLM diagnosis/recovery on failure.
- Data asymmetry: VLMs train on web-scale data (LAION, COCO, Visual Genome — hundreds of millions of pairs). VLAs need synchronized robot demonstration data (camera frames, joint angles, gripper state, success labels) collected via teleoperation, RL, or simulation. Open X-Embodiment is the largest public robot dataset but is far smaller than LAION, limiting VLA generalization. VLAs also demand higher spatial precision from visual encoders (e.g., grasp pose prediction).
- Training: VLMs follow pretrain-then-finetune (often LoRA). VLAs initialize from pre-trained VLMs, extend the action vocabulary, and use action-space regression (or discretized classification) losses, plus Sim2Real transfer, curriculum learning, and multi-task training. Notably, VLM benchmark scores do not predict VLA performance — backbone choice shouldn't rely on VLM leaderboards alone.
- Latency and safety: VLM latency tolerances are loose; VLA control loops run at ~10ms, and 100ms inference destabilizes control (humanoid whole-body control is even more sensitive). VLAs also carry physical failure costs, requiring safety validation layers, action bounds, and recovery mechanisms.
- Applications: VLM alone for recognition/labeling/offline analysis; VLA for end-to-end control (manipulation, humanoid walking, autonomous driving, drone navigation). Most deployed systems use a VLM brain + VLA/controller cerebellum hybrid, e.g., a warehouse robot where the VLM decomposes an instruction into subgoals executed by lower-level policies.
- Evaluation: VLM metrics are mature (VQA accuracy, BLEU/ROUGE, grounding IoU) despite hallucination issues. VLA evaluation needs robustness, safety, and recovery dimensions beyond task success, and requires real robots or high-fidelity simulators.
- Open problems: VLM — hallucination, alignment, fairness (RLHF/DPO, reasoning chains, tool-based fact-checking). VLA — data scarcity, Sim2Real gap, safety verification, and the latency-vs-performance trade-off.
- Notable models: VLMs — CLIP, LLaVA series, Qwen2-VL/Qwen3, Emu3. VLAs — RT-1/RT-2, OpenVLA (2024), GR00T N1 (2025), π0 (2024), plus domain-specific models (CoVLA, OpenDriveVLA for driving).
- Native multimodality is defined by: joint pretraining on interleaved multimodal data from the start; no separate frozen encoder; deep cross-modal attention in all Transformer layers; modality-equal tokens (no "translation" into the LLM's embedding space).
- Gemini (1.0+) is natively multimodal. Its technical report (arXiv:2312.11805) states it trains "natively multimodal models over text, images, audio, and video interleaved in context — from the ground up, not by bolting a frozen vision encoder onto a text decoder." Images become ViT patch tokens, video becomes spatiotemporal patches, audio is discretized to tokens; all tokens flow through a single autoregressive Transformer decoder. Gemini 3.0 adds extreme MoE with fine-grained routing and modality-aware experts. Full architectural details remain undisclosed (closed-source).
- Gemma 3 is NOT native: it uses a frozen 400M-parameter SigLIP encoder (images resized to 896×896) producing 256 soft tokens concatenated with text — essentially the LLaVA recipe (arXiv:2503.19786). The 1B variant has no vision.
- Gemma 4 12B pivots to encoder-free native multimodality (announced June 3, 2026): a ~35M-parameter lightweight embedding module (single matrix multiplication + position embeddings + normalization) replaces the 27-layer ViT; raw 16kHz audio is split into 40ms frames and linearly projected, skipping the 12-layer Conformer; all modalities share identical weights in one decoder-only Transformer. It is Google's first mid-sized model with native audio input, runs in 16GB VRAM, and is Apache 2.0 licensed.
- Three eras of VLM evolution:
- Era 1 (2021–2022): frozen dual towers + learnable bridge (CLIP, BLIP, Flamingo).
- Era 2 (2023–2025): pre-trained LLM as backbone with plug-in visual adapters (LLaVA, Qwen2.5-VL, early GPT-4V).
- Era 3 (2025–2026): no bridge, single tokenizer/embedding space, single Transformer. Era 3a: native multimodal input → text output (Gemini 3, Gemma 4, Qwen3.5, GPT-5.4, Claude Opus 4.6). Era 3b: omni-modal unified input/output with generation heads for images/audio (BAGEL, Emu3/3.5, Janus-Pro, Ernie 5.0).
- Native vs. bolted-on trade-offs: native multimodality gives deep cross-modal fusion, modality equality, and end-to-end optimization, but at very high training cost and without swappable components. Bolted-on designs are cheap to train, modular, and have mature open-source ecosystems — so native is not universally superior.
Part 2: Gemini and Gemma architectures
Conclusion
1. Gemini (1.0+) is genuinely natively multimodal; 2. Gemma 3 is a bolted-on LLM+ViT design; 3. Gemma 4 12B marks Gemma's fundamental shift to encoder-free native multimodality; 4. The 2026 flagship trend (Gemini 3, Gemma 4, Qwen3.5, GPT-5.4) is Era 3a native multimodal input architectures. On the VLA side: most practical robotic systems pair a VLM for perception/planning with a VLA or dedicated controller for execution.
References
1. Gemini 1.0 Technical Report, arXiv:2312.11805, Google DeepMind, 2023 2. Gemma 3 Technical Report, arXiv:2503.19786, Google DeepMind, 2025 3. Gemma 4 12B Announcement, Google Official Blog, 2026-06-03 4. *A Survey of State of the Art Large Vision Language Models*, arXiv:2501.02189, CVPR 2025 Workshop 5. Vision-Language Models Overview: https://github.com/zli12321/Vision-Language-Models-Overview