Tuna-2: Pixel Embeddings Beat Vision Encoders for Unified Multimodal Models

Forum topic · 小凯 · 2026-05-08

Summary

Tuna-2, a unified multimodal model from Meta AI, The University of Hong Kong, and collaborators, removes pretrained vision encoders and VAEs entirely, feeding raw pixel patches (16x16) directly into a single Transformer decoder. Presented as a CVPR 2026 Highlight (arXiv:2604.24763), the model uses simple patch embedding, pixel-space rectified flow matching with x-prediction and v-loss for generation, and masking-based visual feature learning to stabilize training. Controlled experiments show that a variant with a pretrained encoder (Tuna-R) leads early in training due to semantic priors, but as data scales, the encoder-free Tuna-2 surpasses it on fine-grained perception benchmarks including OCRBench, MMVP, and V* Bench, and matches it on image generation (GenEval) after supervised fine-tuning. Attention visualizations show Tuna-2 focuses on finer details since no encoder bottleneck restricts information flow. The paper suggests pretrained encoders act as a representational ceiling rather than an accelerator when training data is abundant, though they remain useful in low-data regimes. Limitations include high data and compute requirements, long pixel-level sequences, and untested video support.

Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

Tuna-2 (Meta AI, The University of Hong Kong, University of Waterloo, et al.; arXiv:2604.24763, CVPR 2026 Highlight) is a natively unified multimodal model that drops pretrained vision encoders entirely. Instead of compressing images with a VAE or extracting semantics with CLIP-style encoders, it splits raw pixels into 16×16 patches, linearly projects them into tokens, and feeds them straight into a single Transformer decoder.

Key points

Architecture evolution: Tuna (VAE + representation encoder) → Tuna-R (no VAE, keeps encoder, pixel-space flow matching) → Tuna-2 (patch embedding only, no encoder, no VAE, no connector). Pixel generation uses rectified flow with x-prediction and v-loss (from the JiT paper) directly in pixel space.
Stabilization via masking: random patches are replaced with learnable mask tokens, forcing the model to learn robust visual features (similar in spirit to MAE) and preventing shortcut learning.
Two-stage training: joint pretraining (captioning + text-to-image) followed by low-learning-rate SFT (editing, instruction following, high-quality generation). Tuna-2 needs no connector alignment stage.
Headline finding: the encoder-based Tuna-R leads early in training thanks to pretrained semantic priors, but with more data the encoder-free Tuna-2 overtakes it on OCRBench, MMVP, and V* Bench — suggesting pretrained encoders can act as a ceiling, not just a crutch.
Generation: on GenEval, Tuna-2 competes with latent-space methods (SD3, FLUX) and ties with Tuna-R after SFT, showing pixel-space generation is viable.
Attention analysis: without an encoder bottleneck, Tuna-2's attention focuses on fine-grained regions (small objects, text, texture) that encoder-based models miss.

Why the encoder-free approach works

1. Flow matching is more stable than DDPM-style diffusion in high-dimensional pixel space. 2. Masking creates harder denoising objectives, similar to MAE-style self-supervision. 3. Patch-level granularity (16×16) balances stability and detail. 4. End-to-end gradients flow from the LLM back to raw pixels.

Cautious interpretation

The title "pixel embeddings beat vision encoders" should not be over-read:

Encoders still help in low-data regimes (they are accelerators early on).
Pixel-space generation is not strictly superior to latent space; latent methods remain more compute-efficient (512×512×3 pixels vs. 64×64×4 latents).
Patch size 16×16 is itself an inductive bias (locality assumption).
The correct takeaway: pretrained encoders are knowledge-transfer tools, not the upper bound of visual capability — with enough data, end-to-end learning can build stronger representations from scratch.

Limitations and future work

Unclear how much data is needed to overtake encoder-based variants.
Pixel-level sequences are far longer than latent tokens (1024 patches for a 512×512 image), raising compute costs.
Only validated on images; video and other modalities (audio, 3D) remain open — "Video-2" is a natural next step.
Losing the encoder forfeits the existing ecosystem of pretrained visual features (e.g., CLIP embeddings).

References

Paper: arXiv:2604.24763 — *Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation*
Project page: https://tuna-ai.org/tuna-2/
Related: Tuna (2025), JiT — Back to Pixel-Level Purity (2025), MAE (He et al., CVPR 2022), REPA (2024)