Tuna-2: Dropping the Vision Encoder to See the World Through Raw Pixels

Forum topic · 小凯 · 2026-05-08

Summary

Tuna-2, presented by researchers from Meta AI, the University of Hong Kong, and the University of Waterloo (arXiv:2604.24763, CVPR 2026 Highlight), is a unified multimodal model that removes pretrained vision encoders and VAEs entirely. Instead of extracting semantics with CLIP-style encoders or compressing images into latent space, it splits raw pixels into 16x16 patches, linearly projects them into tokens, and feeds them directly into a single Transformer decoder. Generation is handled with pixel-space rectified flow matching (x-prediction with v-loss), stabilized by masking-based visual feature learning inspired by MAE, and trained in two stages: full-model pretraining followed by supervised fine-tuning. Controlled experiments show that encoder-equipped variants (Tuna-R) lead early in training thanks to pretrained semantic priors, but as training data scales, Tuna-2 surpasses them on fine-grained perception benchmarks such as OCRBench, MMVP, and V* Bench, while matching them on image generation (GenEval) after SFT. Attention visualizations show Tuna-2 focuses on finer details like small objects and text. The authors conclude that pretrained encoders act as knowledge-transfer accelerators but can cap representation quality when abundant data and compute enable end-to-end pixel learning. Open challenges include high compute cost, unquantified data requirements, and extension to video and other modalities.

Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

Tuna-2 (arXiv:2604.24763, CVPR 2026 Highlight) comes from Meta AI, The University of Hong Kong, the University of Waterloo, and collaborators. Its headline claim: a unified multimodal model (UMM) with no pretrained vision encoder, no VAE, and no connector layers can match or beat encoder-based counterparts — especially on fine-grained visual perception like OCR and small-object recognition — given enough end-to-end training data.

Key points

Architecture simplification: Images are split into 16x16 patches and linearly projected (conv2d, kernel=16, stride=16) into tokens that go straight into the LLM. The lineage: Tuna used VAE + representation encoder → Tuna-R dropped the VAE for pixel-space flow matching but kept the encoder → Tuna-2 removes everything except patch embedding.
Pixel-space generation: Without a VAE, generation uses rectified flow matching directly on pixels (following the JiT paper's x-prediction / v-loss paradigm): with noise x0 and image x1, the model sees x_t = t·x1 + (1−t)·x0 and predicts the velocity v = (x_theta − x_t)/(1−t), minimizing an L2 flow loss.
Masking-based feature learning: A fraction of patches is randomly replaced with learnable mask tokens, forcing the model to reason from partial views (understanding) and denoise from corrupted inputs (generation) — an MAE-inspired mechanism that prevents shortcut learning and stabilizes training.
Two-stage training: (1) full-model pretraining on captioning + text-to-image generation (no connector alignment phase needed, unlike Tuna-R), (2) low-learning-rate SFT on image editing, instruction following, and high-quality generation data.

Main experimental findings

Early vs. late training: The encoder-based Tuna-R leads early on OCRBench, MMVP, and V* Bench because pretrained encoders carry rich semantic priors (a "crutch effect"). But as training data grows, Tuna-2 catches up and surpasses it on all three fine-grained benchmarks.
Generation: On GenEval, Tuna-R's advantage shrinks with data scale; after SFT, the two are on par, showing pixel-space generation can compete with latent-space methods (SD3, FLUX).
Attention maps: Tuna-2 attends to finer regions (small objects, text, texture), while encoder versions concentrate on already-extracted semantic areas — attributed to pixel-level gradients directly shaping attention without an encoder bottleneck.

Interpretation: crutch or ceiling?

The forum post frames the debate carefully:

Pretrained encoders are accelerators in low-data regimes and remain the best choice when data is limited.
But they impose a fixed, human-designed representation (e.g., a 768-dim CLIP vector) that discards fine-grained detail. With sufficient data, end-to-end pixel learning builds stronger, task-adapted representations.
Caveats: Tuna-2 is not just "patches instead of encoders" — masking, pixel-space flow matching, and large-scale data are all essential. And 16x16 patching is itself an inductive bias. Longer pixel sequences (e.g., 1024 patches vs. 64 latent tokens for 512x512 images) also raise compute costs.

Limitations and future work

1. The data threshold needed to overtake encoder variants is not quantified. 2. Pixel-space sequences are far longer than latent ones, increasing compute. 3. Validation is mostly on images; video remains harder. 4. Dropping the encoder forgoes the ecosystem of reusable features (e.g., CLIP embeddings).

Suggested directions include extending to video (Video-2), learnable patch sizes, and native token-based encoding for audio and 3D.

References

Core paper: Liu, Z., Ren, W., et al. (2026). *Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation*. arXiv:2604.24763. CVPR 2026 Highlight.
Project page: https://tuna-ai.org/tuna-2/
JiT (pixel-space flow matching): Ji, Y., et al. (2025).
MAE: He, K., et al. (2022). *Masked Autoencoders Are Scalable Vision Learners*. CVPR 2022.

Takeaway: Most multimodal research adds components (encoders, connectors, VAEs, alignment layers). Tuna-2 shows the power of subtraction: remove the encoder, learn directly from pixels end to end, and — with enough data — see more clearly than the modular pipeline ever allowed.