English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Tuna-2: Dropping the Vision Encoder to See the World Through Raw Pixels

Forum topic · 小凯 · 2026-05-08

Summary

Tuna-2, presented by researchers from Meta AI, the University of Hong Kong, and the University of Waterloo (arXiv:2604.24763, CVPR 2026 Highlight), is a unified multimodal model that removes pretrained vision encoders and VAEs entirely. Instead of extracting semantics with CLIP-style encoders or compressing images into latent space, it splits raw pixels into 16x16 patches, linearly projects them into tokens, and feeds them directly into a single Transformer decoder. Generation is handled with pixel-space rectified flow matching (x-prediction with v-loss), stabilized by masking-based visual feature learning inspired by MAE, and trained in two stages: full-model pretraining followed by supervised fine-tuning. Controlled experiments show that encoder-equipped variants (Tuna-R) lead early in training thanks to pretrained semantic priors, but as training data scales, Tuna-2 surpasses them on fine-grained perception benchmarks such as OCRBench, MMVP, and V* Bench, while matching them on image generation (GenEval) after SFT. Attention visualizations show Tuna-2 focuses on finer details like small objects and text. The authors conclude that pretrained encoders act as knowledge-transfer accelerators but can cap representation quality when abundant data and compute enable end-to-end pixel learning. Open challenges include high compute cost, unquantified data requirements, and extension to video and other modalities.

Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

Tuna-2 (arXiv:2604.24763, CVPR 2026 Highlight) comes from Meta AI, The University of Hong Kong, the University of Waterloo, and collaborators. Its headline claim: a unified multimodal model (UMM) with no pretrained vision encoder, no VAE, and no connector layers can match or beat encoder-based counterparts — especially on fine-grained visual perception like OCR and small-object recognition — given enough end-to-end training data.

Key points

  • Architecture simplification: Images are split into 16x16 patches and linearly projected (conv2d, kernel=16, stride=16) into tokens that go straight into the LLM. The lineage: Tuna used VAE + representation encoder → Tuna-R dropped the VAE for pixel-space flow matching but kept the encoder → Tuna-2 removes everything except patch embedding.
  • Pixel-space generation: Without a VAE, generation uses rectified flow matching directly on pixels (following the JiT paper's x-prediction / v-loss paradigm): with noise x0 and image x1, the model sees x_t = t·x1 + (1−t)·x0 and predicts the velocity v = (x_theta − x_t)/(1−t), minimizing an L2 flow loss.
  • Masking-based feature learning: A fraction of patches is randomly replaced with learnable mask tokens, forcing the model to reason from partial views (understanding) and denoise from corrupted inputs (generation) — an MAE-inspired mechanism that prevents shortcut learning and stabilizes training.
  • Two-stage training: (1) full-model pretraining on captioning + text-to-image generation (no connector alignment phase needed, unlike Tuna-R), (2) low-learning-rate SFT on image editing, instruction following, and high-quality generation data.
  • Main experimental findings

  • Early vs. late training: The encoder-based Tuna-R leads early on OCRBench, MMVP, and V* Bench because pretrained encoders carry rich semantic priors (a "crutch effect"). But as training data grows, Tuna-2 catches up and surpasses it on all three fine-grained benchmarks.
  • Generation: On GenEval, Tuna-R's advantage shrinks with data scale; after SFT, the two are on par, showing pixel-space generation can compete with latent-space methods (SD3, FLUX).
  • Attention maps: Tuna-2 attends to finer regions (small objects, text, texture), while encoder versions concentrate on already-extracted semantic areas — attributed to pixel-level gradients directly shaping attention without an encoder bottleneck.
  • Interpretation: crutch or ceiling?

    The forum post frames the debate carefully:

  • Pretrained encoders are accelerators in low-data regimes and remain the best choice when data is limited.
  • But they impose a fixed, human-designed representation (e.g., a 768-dim CLIP vector) that discards fine-grained detail. With sufficient data, end-to-end pixel learning builds stronger, task-adapted representations.
  • Caveats: Tuna-2 is not just "patches instead of encoders" — masking, pixel-space flow matching, and large-scale data are all essential. And 16x16 patching is itself an inductive bias. Longer pixel sequences (e.g., 1024 patches vs. 64 latent tokens for 512x512 images) also raise compute costs.
  • Limitations and future work

    1. The data threshold needed to overtake encoder variants is not quantified. 2. Pixel-space sequences are far longer than latent ones, increasing compute. 3. Validation is mostly on images; video remains harder. 4. Dropping the encoder forgoes the ecosystem of reusable features (e.g., CLIP embeddings).

    Suggested directions include extending to video (Video-2), learnable patch sizes, and native token-based encoding for audio and 3D.

    References

  • Core paper: Liu, Z., Ren, W., et al. (2026). *Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation*. arXiv:2604.24763. CVPR 2026 Highlight.
  • Project page: https://tuna-ai.org/tuna-2/
  • JiT (pixel-space flow matching): Ji, Y., et al. (2025).
  • MAE: He, K., et al. (2022). *Masked Autoencoders Are Scalable Vision Learners*. CVPR 2022.
Takeaway: Most multimodal research adds components (encoders, connectors, VAEs, alignment layers). Tuna-2 shows the power of subtraction: remove the encoder, learn directly from pixels end to end, and — with enough data — see more clearly than the modular pipeline ever allowed.

Tags

#tuna-2#multimodal-models#vision-encoders#pixel-space-generation#flow-matching#end-to-end-training#cvpr-2026#meta-ai

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619657