Tuna-2: Pixel Embeddings Beat Vision Encoders — An End-to-End Native Unified Multimodal Model

Forum topic · 小凯 · 2026-05-08

Summary

Tuna-2, a unified multimodal model from Meta AI, The University of Hong Kong, and University of Waterloo (arXiv:2604.24763, CVPR 2026 Highlight), removes pretrained vision encoders and VAEs entirely. Raw pixels are split into 16x16 patches, linearly projected, and fed directly into a single Transformer decoder, enabling fully end-to-end learning with pixel-space flow matching (rectified flow with x-prediction and v-loss) and masking-based visual feature learning for stable training. Controlled experiments show that the encoder-based variant (Tuna-R) leads early in training thanks to pretrained semantic priors, but Tuna-2 catches up and surpasses it on fine-grained perception benchmarks like OCRBench, MMVP, and V* Bench as training data scales, and reaches parity on generation (GenEval) after SFT. Attention visualizations confirm Tuna-2 focuses on finer details without the encoder information bottleneck. The post analyzes whether pretrained encoders act as a crutch or a ceiling, notes limitations (data requirements, compute cost of long pixel sequences, video extension), and frames the result as evidence that end-to-end pixel learning can exceed encoder representations given sufficient data.

Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

> TL;DR: Tuna-2 (Meta AI, HKU, Waterloo et al.; arXiv:2604.24763; CVPR 2026 Highlight) is a unified multimodal model that uses no pretrained vision encoder and no VAE. Raw pixels are split into patches and fed straight into a Transformer — and on fine-grained perception tasks (OCR, small-object recognition), it surpasses encoder-based counterparts. Encoder variants lead early in training (the "crutch effect" of priors), but end-to-end pixel learning overtakes them as data scales — suggesting pretrained encoders may be a ceiling, not just an accelerator.

Key points

Architecture evolution
Tuna (previous): image → VAE + representation encoder → LLM
Tuna-R (intermediate): drops VAE, keeps representation encoder, adds pixel-space flow matching
Tuna-2: image → simple patch embedding (16x16 patches, conv2d k=16, s=16) → LLM. No encoder, no VAE, no connector.
Problems with the modular pipeline (encoder → connector → LLM, VAE for generation):
Two misaligned visual representations (semantic vectors for understanding vs. VAE latents for generation)
Broken end-to-end gradients: LLM learns from the encoder's "second-hand" features
Information bottleneck: CLIP-style compression loses fine-grained detail (small text, object counts)
Enabling techniques
Pixel-space flow matching: rectified flow with x-prediction and v-loss (from JiT), constructing $x_t = t \cdot x_1 + (1-t) \cdot x_0$ and minimizing $\mathcal{L}_{flow} = \mathbb{E} ||v_\theta - v||_2^2$
Masking-based visual feature learning: random patches replaced with learnable mask tokens, forcing harder denoising and preventing shortcuts (MAE-like, extended to unified understanding + generation)
Two-stage training: joint captioning + text-to-image pretraining (no connector-alignment stage needed, unlike Tuna-R), then low-LR SFT on editing / instruction following / high-quality generation
Results
Encoder version (Tuna-R) leads early on OCRBench, MMVP, V* Bench; Tuna-2 surpasses it at scale
Generation: encoder advantage on GenEval fades with data; near parity after SFT, competing with latent-space models (SD3, FLUX)
Attention maps: Tuna-2 attends to finer regions (small objects, text, textures) because pixel-level gradients aren't filtered through an encoder bottleneck

Analysis: crutch or ceiling?

Crutch view: encoders carry priors (e.g., CLIP trained on 400M pairs) and remain optimal when data is limited
Ceiling view: fixed, human-designed representation spaces discard fine-grained information; end-to-end learning can build task-adapted representations from scratch
Tuna-2's experiments support the ceiling view at large data scales — but it is not a naive substitution: masking, pixel-space flow matching, and large-scale data are all required

Avoiding cargo-cult readings

❌ "All models should drop encoders" — with limited data, encoders still accelerate
❌ "Pixel-space generation is better than latent" — Tuna-2 shows competitiveness; latents retain compute-efficiency advantages
❌ "Hand-designed inductive biases are useless" — a 16x16 patch size is itself a locality prior
✅ Correct takeaway: pretrained encoders are knowledge-transfer tools, not cognitive limits; with enough data, end-to-end learning can break through their representation ceiling

Layman's explanation

> Today's AI sees images in two steps: a "translator" (encoder) compresses the image into a summary of a few hundred numbers, which the brain (LLM) reads. Tuna-2 lets the brain look at the raw image directly, in 16x16-pixel squares. Early on, the version with the translator wins — it has learned from 400M images. But after seeing enough data, Tuna-2 sees finer detail: the translator sacrificed detail for generality, while Tuna-2 optimizes every pixel for the task. Hiring an experienced translator vs. learning the language yourself: the former starts faster, the latter has a higher ceiling.

Limitations and future work

Unspecified threshold of "enough" data; high compute cost (512x512 image → 1024 patches vs. 64 tokens in latent space)
Validated mainly on images; video (spatiotemporal modeling) is harder
Loses compatibility with encoder-feature ecosystems (e.g., CLIP embeddings)
Future directions: video extension ("Video-2"), interaction with T² scaling laws, learnable patch sizes, native tokenization for audio/3D/touch

References

Core paper: Liu, Z., Ren, W., et al. (2026). *Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation*. arXiv:2604.24763. CVPR 2026 Highlight.
Project page: https://tuna-ai.org/tuna-2/
Tuna (prior work): Liu, Z., et al. (2025). *Tuna: Unified Visual Representation for Native Multimodal Models*
JiT (flow matching): Ji, Y., et al. (2025). *JiT: Back to Pixel-Level Purity*
MAE: He, K., et al. (2022). *Masked Autoencoders Are Scalable Vision Learners*. CVPR 2022.
REPA: Yu, L., et al. (2024). *REPA: Representation Alignment for Generation*

Closing thought: The multimodal field has been adding components (encoders, connectors, VAEs, alignment layers). Tuna-2 does subtraction instead — and wins. But be careful not to over-read: Tuna-2 was weaker early on, and only compute-rich labs can afford the scale where dropping the encoder pays off. The open question is whether this "native unified" recipe extends to 3D, audio, and other modalities.