Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation
Tuna-2 (Meta AI, The University of Hong Kong, University of Waterloo, et al.; arXiv:2604.24763, CVPR 2026 Highlight) is a natively unified multimodal model that drops pretrained vision encoders entirely. Instead of compressing images with a VAE or extracting semantics with CLIP-style encoders, it splits raw pixels into 16×16 patches, linearly projects them into tokens, and feeds them straight into a single Transformer decoder.
Key points
- Architecture evolution: Tuna (VAE + representation encoder) → Tuna-R (no VAE, keeps encoder, pixel-space flow matching) → Tuna-2 (patch embedding only, no encoder, no VAE, no connector). Pixel generation uses rectified flow with x-prediction and v-loss (from the JiT paper) directly in pixel space.
- Stabilization via masking: random patches are replaced with learnable mask tokens, forcing the model to learn robust visual features (similar in spirit to MAE) and preventing shortcut learning.
- Two-stage training: joint pretraining (captioning + text-to-image) followed by low-learning-rate SFT (editing, instruction following, high-quality generation). Tuna-2 needs no connector alignment stage.
- Headline finding: the encoder-based Tuna-R leads early in training thanks to pretrained semantic priors, but with more data the encoder-free Tuna-2 overtakes it on OCRBench, MMVP, and V* Bench — suggesting pretrained encoders can act as a ceiling, not just a crutch.
- Generation: on GenEval, Tuna-2 competes with latent-space methods (SD3, FLUX) and ties with Tuna-R after SFT, showing pixel-space generation is viable.
- Attention analysis: without an encoder bottleneck, Tuna-2's attention focuses on fine-grained regions (small objects, text, texture) that encoder-based models miss.
- Encoders still help in low-data regimes (they are accelerators early on).
- Pixel-space generation is not strictly superior to latent space; latent methods remain more compute-efficient (512×512×3 pixels vs. 64×64×4 latents).
- Patch size 16×16 is itself an inductive bias (locality assumption).
- The correct takeaway: pretrained encoders are knowledge-transfer tools, not the upper bound of visual capability — with enough data, end-to-end learning can build stronger representations from scratch.
- Unclear how much data is needed to overtake encoder-based variants.
- Pixel-level sequences are far longer than latent tokens (1024 patches for a 512×512 image), raising compute costs.
- Only validated on images; video and other modalities (audio, 3D) remain open — "Video-2" is a natural next step.
- Losing the encoder forfeits the existing ecosystem of pretrained visual features (e.g., CLIP embeddings).
- Paper: arXiv:2604.24763 — *Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation*
- Project page: https://tuna-ai.org/tuna-2/
- Related: Tuna (2025), JiT — Back to Pixel-Level Purity (2025), MAE (He et al., CVPR 2022), REPA (2024)
Why the encoder-free approach works
1. Flow matching is more stable than DDPM-style diffusion in high-dimensional pixel space. 2. Masking creates harder denoising objectives, similar to MAE-style self-supervision. 3. Patch-level granularity (16×16) balances stability and detail. 4. End-to-end gradients flow from the LLM back to raw pixels.
Cautious interpretation
The title "pixel embeddings beat vision encoders" should not be over-read: