Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding

Forum topic · 小凯 · 2026-04-29

Summary

Tuna-2 is a native unified multimodal model that performs both visual understanding and generation directly on pixel embeddings, eliminating modular vision encoders such as VAEs or representation encoders. Instead of relying on pretrained vision encoders and separate visual representations for understanding and generation— which causes task misalignment and blocks end-to-end optimization from raw pixels—Tuna-2 uses simple patch embedding layers to encode visual input, drastically simplifying the architecture. Experiments show that Tuna-2 achieves state-of-the-art performance on multimodal benchmarks, demonstrating that unified pixel-space modelling can fully compete with latent-space approaches for high-quality image generation. While encoder-based variants converge faster during early pretraining, Tuna-2's encoder-free design achieves stronger multimodal understanding at scale, particularly on tasks requiring fine-grained visual perception. The paper (arXiv:2504.20693) by Zhiheng Liu, Weiming Ren, and Xiaoke Huang was released on April 29, 2025.

论文概要

Research area: Computer Vision Authors: Zhiheng Liu, Weiming Ren, Xiaoke Huang Published: 2025-04-29 arXiv: 2504.20693

Abstract (translated)

Tuna-2 is a native unified multimodal model that performs visual understanding and generation directly on pixel embeddings, completely abandoning modular vision encoder designs such as VAEs or representation encoders. Experiments show that Tuna-2 achieves SOTA performance on multimodal benchmarks, demonstrating that unified pixel-space modeling can compete with latent-space approaches for high-quality image generation. Although encoder-based variants converge faster in early pretraining, Tuna-2's encoder-free design achieves stronger multimodal understanding at scale, especially on tasks requiring fine-grained visual perception.

Original Abstract

Unified multimodal models typically rely on pretrained vision encoders and use separate visual representations for understanding and generation, creating misalignment between the two tasks and preventing fully end-to-end optimization from raw pixels. We introduce Tuna-2, a native unified multimodal model that performs visual understanding and generation directly based on pixel embeddings. Tuna-2 drastically simplifies the model architecture by employing simple patch embedding layers to encode visual input, completely discarding the modular vision encoder designs such as the VAE or the representation encoder. Experiments show that Tuna-2 achieves state-of-the-art performance in multimodal benchmarks, demonstrating that unified pixel-space modelling can fully compete with latent-space approaches.

Key Takeaways

Encoder-free architecture: Simple patch embedding layers replace VAE and representation encoders.
Unified pixel space: Understanding and generation share a single pixel-based representation, avoiding task misalignment.
SOTA results: Competitive or superior performance on multimodal benchmarks versus latent-space methods.
Scaling insight: Slower early convergence than encoder-based variants, but stronger fine-grained understanding at scale.

---

*Auto-collected on 2026-04-29*