VLA vs VLM and Gemini/Gemma Architecture: A Comparative Technical Survey

Forum topic · 小凯 · 2026-06-15

Summary

This two-part technical report compares Vision-Language Models (VLM) and Vision-Language-Action Models (VLA), then investigates the architectures of Google's Gemini and Gemma model families. Part one clarifies that VLMs output text while VLAs output actions, surveying architectural variants (end-to-end, dual-system, hierarchical, self-correcting), training data scarcity, evaluation challenges, and deployment constraints such as millisecond-level latency. Part two defines native multimodality and shows Gemini 1.0-3.0 is natively multimodal, Gemma 3 uses a frozen SigLIP encoder with an LLM backbone (LLM+ViT style), while Gemma 4 12B (June 2026) adopts an encoder-free native multimodal architecture that needs only 16GB of memory. It maps the VLM evolution across three eras and concludes that encoder-free native multimodal designs now dominate flagship models.

Key Points

Part 1 — VLA vs VLM: Deep Comparison

Core distinction: VLMs output text; VLAs output actions (motor commands, joint angles, control signals).
VLM architecture lineage: From CLIP contrastive learning to Flamingo/BLIP generative models, then LLaVA's "ViT + projection + pretrained LLM" template adopted by Qwen2-VL and LLaMA 3.2 Vision. Emu3 explores token-level unified multimodal processing.
VLA architecture variants:
*End-to-end*: RT-1, RT-2 (Google), OpenVLA — simple but limited cross-robot generalization.
*Dual-system*: NVIDIA GR00T N1 — fast reactive "System 1" (~10ms latency) plus slower reasoning "System 2".
*Hierarchical*: CogACT, NaVILA — upper LLM for planning, lower controllers for execution; modular but latency accumulates.
*Self-correcting*: SC-VLA — invokes an LLM for diagnosis and recovery on failure.
Data asymmetry: VLMs train on billion-scale web datasets (LAION, COCO, Visual Genome). VLAs depend on expensive robot demonstrations; Open X-Embodiment is large but still small relative to LAION. VLA visual encoders also demand higher spatial precision for grasp-pose prediction.
Training methods: VLMs use pretraining + LoRA-style fine-tuning. VLAs initialize from a pretrained VLM, extend the action vocabulary, and use regression or discretized classification losses plus Sim2Real, curriculum learning, and multi-task training. Notably, VLM benchmark scores do not predict VLA performance.
Deployment constraints: VLA control loops run at ~10ms; model inference latency and jitter directly affect stability for humanoids. Common acceleration techniques include distillation, quantization, early exit, and lightweight backbones. Failure cost is far higher for VLAs, requiring safety alignment, action-bound limits, and recovery mechanisms.
Application split: VLM-only for perception, captioning, semantic annotation; VLA for end-to-end control (manipulation, humanoid locomotion, end-to-end driving, UAV navigation). Most real systems use VLM-as-brain + VLA/controller-as-cerebellum hybrids.
Evaluation gap: VLM uses VQA accuracy, BLEU/ROUGE, IoU. VLA evaluation now stresses robustness, safety, recovery ability, and execution efficiency, requiring real-robot or high-fidelity simulation testing.
Open problems: VLMs face hallucination and alignment; VLAs face data scarcity, Sim2Real gap, safety verification, and latency–performance trade-offs.
Representative models: VLMs — CLIP, LLaVA family, Qwen2-VL/Qwen3, Emu3. VLAs — RT-1/RT-2, OpenVLA, GR00T N1, Pi-0, plus domain-specific variants in driving, medical, and agricultural robotics.

Part 2 — Gemini and Gemma Architecture Survey

Definition of native multimodality: Joint pretraining from scratch on interleaved text/image/audio/video; no separately pretrained and frozen encoder; deep cross-modal attention at every Transformer layer; modality-equivalent tokens. The opposite is the LLaVA-style "frozen ViT + projection + pretrained LLM" glued design.
Gemini (1.0 → 3.0) is native multimodal:
Gemini 1.0 technical report explicitly states it is trained natively multimodal, *not* by bolting a frozen vision encoder onto a text decoder.
Text uses BPE tokens; images become ViT patch tokens (e.g., 256 tokens for a 256×256 image); audio is discretized via VQ-VAE / USM-style encoders; video is split into spatiotemporal patches rather than treated as independent frames.
All modalities enter a single Transformer decoder, identified only by modality embeddings; interleaved sequences like [text, image, video, text, audio] are supported.
Unified autoregressive objective across mixed token sequence $(z_1, z_2, \ldots, z_T)$.
Gemini 3.0 uses fine-grained routed MoE with modality-aware experts for vision, audio, and code.
Gemma 3 is not native multimodal:
Uses a 400M-parameter frozen SigLIP ViT encoder (images resized to 896×896, encoded into 256 soft tokens).
4B/12B/27B share the frozen encoder; the 1B variant has no vision capability.
Architecture is essentially LLaVA-style: frozen SigLIP + projection + pretrained LLM.
Gemma 4 12B (2026-06-03) is encoder-free native multimodal:
Replaces the vision encoder with a 35M-parameter embedding module (single matmul + positional embedding + normalization).
Drops the 12-layer Conformer audio encoder; raw 16kHz audio is split into 40ms frames and linearly projected to the LLM input space.
Vision, audio, and text share identical backbone weights; first mid-sized Google model with native audio input; runs on 16GB of memory.
Three eras of VLM architecture:
*Era 1 (2021–2022)*: Dual-tower + learnable bridges (Q-Former). Examples: CLIP, BLIP, Flamingo.
*Era 2 (2023–2025)*: Pretrained LLM as backbone with plug-in vision adapters. Examples: LLaVA, Qwen2.5-VL, early GPT-4V.
*Era 3 (2025–2026)*: No bridge module; shared tokenizer/embedding; single Transformer trained from scratch. Branches into:
*Era 3a — native multimodal input → text output*: Qwen3.5/3.6, Gemma 4, Gemini 3, GPT-5.4, Phi-4-Reasoning-Vision, Claude Opus 4.6.
*Era 3b — omni-modal unified I/O*: Adds dedicated decoder heads (image VAE/MMDiT/flow-matching, audio codec heads). Examples: BAGEL, Qwen3.5-Omni, InternVL-U, Emu3/3.5, Janus-Pro, ERNIE 5.0.
Trade-offs: Native multimodal architectures enable deeper cross-modal fusion and modality-equal treatment, but training cost is high and components cannot be independently swapped. LLM+ViT designs reuse pretrained modules, train cheaply, allow independent upgrades, and benefit from mature open-source ecosystems.

Conclusion

Gemini (1.0+) is native multimodal.
Gemma 3 is an LLM+ViT glued architecture (similar to LLaVA).
Gemma 4 12B is native multimodal via an encoder-free design — a fundamental turning point for the Gemma family and a reflection of broader industry direction.
The 2026 flagship cohort (Gemini 3, Gemma 4, Qwen3.5, GPT-5.4, etc.) has converged on Era 3a native multimodal input architectures.

References

1. Gemini 1.0 Technical Report: *Gemini: A Family of Highly Capable Multimodal Models*, arXiv:2312.11805, Google DeepMind, 2023. 2. Gemma 3 Technical Report, arXiv:2503.19786, Google DeepMind, 2025. 3. Gemma 4 12B Announcement, Google Official Blog, 2026-06-03. 4. *A Survey of State of the Art Large Vision Language Models*, arXiv:2501.02189, CVPR 2025 Workshop. 5. Vision-Language Models Overview: https://github.com/zli12321/Vision-Language-Models-Overview