Dropping the Encoder! Tuna-2 Declares: The Vision Encoder Is Just "Scaffolding," Pixels Are Justice
Introduction: If you wanted to teach a blind person to appreciate a painting, would you hire a translator to describe the image in words — or let them touch every texture and line directly with their fingertips?
In multimodal AI, we have long relied on expensive "translators" — vision encoders (like CLIP or VAE). But Meta's latest paper "Tuna-2" argues the translator can retire. Letting the model directly "touch" raw pixels may be the ultimate path toward general-purpose AI.
---
#### 1. The Rejected "Translator": What Is Representation Misalignment?
Traditionally, we either use CLIP to teach a model to "look and describe," or a VAE to teach it to "generate images." The problem: these two translators speak different dialects.
This causes representation misalignment — the model uses one logic for understanding and another for generation. The result is a Frankenstein multimodal model where perception and generation are never quite on the same channel.
Worse, these translators "smooth out" details. Ask how many buttons are in the picture, and you may just hear "a person wearing a shirt."
#### 2. "Pixels Are Justice": Tearing Down the Scaffolding
Tuna-2's core trick is simple and aggressive: completely remove the vision encoder.
It uses pixel embeddings directly.
Feynman-style analogy: It's like cutting a photograph into tiny fragments and feeding them straight to the model. Instead of hearing a translator's retelling, the model faces the rawest, most fundamental pixels.
Benefits:
- Every detail visible: With no middleman, the model can perceive extremely fine details. On counting and tiny-object recognition tasks, Tuna-2 outperforms established large models.
- One unified language: Understanding and generation share a single "pixel logic," achieving native cross-modal alignment without complex alignment training.
Why wasn't this done before? Answer: compute was insufficient, and no good learning method existed.
Tuna-2 introduces flow matching, giving the model a pair of eyes that can see through noise, letting it reliably generate high-quality images directly in raw pixel space.
Experiments show encoder-based models converge faster early in training, but with enough data and compute, the encoder-free Tuna-2 has a higher ceiling and stronger long-term potential.
---
#### Editorial Commentary
Tuna-2's success forces a reflection: were our painstakingly designed complex architectures merely "scaffolding" from multimodal AI's early days?
As we pursued higher-level abstractions, did we inadvertently discard the most authentic foundation? AI architecture seems to be undergoing a return to simplicity: from complex feature engineering back to raw pixels, from stacked modules back to a single Transformer.
If "de-encoderization" becomes mainstream, what will be the next piece of "scaffolding" to be torn down in AI? Share your predictions in the comments!
--- *Note: Based on a recent Meta Research paper, representing the frontier of native multimodal modeling.*