Image Generators as Generalist Vision Learners: When AI Image Generators Develop 'Golden Eyes'

Forum topic · 小凯 · 2026-05-22

Summary

A deep dive into the paper 'Image Generators are Generalist Vision Learners' by He Kaiming and Google Research (2026), which shows that the Vision Banana image generator can serve as a universal vision model. Instead of training separate models for depth estimation or semantic segmentation, all perception tasks are reformulated as image generation: the model 'paints' depth maps and segmentation masks. A log-based non-linear mapping compresses real-world distances into RGB values, emphasizing nearby detail. Despite being trained purely on synthetic data, Vision Banana beats specialist models like SAM 3, Depth Anything V3, Lotus-2, and MoGe-2 on benchmarks including ADE20K segmentation (mIoU), metric depth (0.929 delta-1 accuracy on NYU, ETH3D, KITTI), and surface normal estimation. Built on Google DeepMind's Nano Banana Pro diffusion engine with lightweight instruction tuning, the work suggests generation and perception share one foundation, pointing toward general world models for robotics and autonomous driving.

Image Generators as Generalist Vision Learners: When AI Image Generators Develop "Golden Eyes"

AI image generation has long been praised for its ability to create something from nothing. But is photorealism the whole story? In April 2026, Google Research together with He Kaiming and collaborators released the paper *Image Generators are Generalist Vision Learners*, arguing that the true ambition of generative vision models lies not in painting, but in understanding the physical world.

Painting Instead of Perceiving: A New Paradigm

The model, Vision Banana, looks like an image generator but functions as a generalist vision system. Its core idea is elegant: rather than building separate specialized models for tasks like depth estimation or semantic segmentation, all perception tasks are reformulated as image generation.

Want a depth map? The model *paints* one.
Want semantic segmentation? The model *paints* a segmentation map.

> What is Generalist Vision Learning? > A paradigm where a model is not trained for a single specialized task (e.g., face recognition only), but instead learns general features from large-scale data so it can handle any visual problem.

The Math: Non-linear Depth Mapping

How do you encode continuous real-world depth into pixels? Vision Banana embeds physical distance $$D$$ into RGB values using:

d_{norm} = C \cdot \log(1 + \lambda \cdot D)

The logarithmic transform stretches near-field detail — mirroring how biological vision prioritizes nearby objects — while an inverse function recovers exact physical distances from the generated image.

> Non-linear Depth Mapping: A technique that compresses unbounded real-world distances into the 0–255 pixel range, using logarithms to focus the model's sensitivity on close-range geometry.

Head-to-Head: Generalist Beats Specialists

On multiple benchmarks, this generation-based generalist outperformed specialist models:

| Task | Metric | Vision Banana (generalist) | Domain Specialist | |---|---|---|---| | Semantic segmentation (ADE20K) | mIoU (higher is better) | Best in class | SAM 3 (outperformed) | | Metric depth estimation | $\delta_1$ accuracy | 0.929 | Depth Anything V3 (0.918) | | Surface normal estimation | Mean angular error (lower is better) | Highly precise | Lotus-2 (less accurate) |

Remarkably, Vision Banana's 3D understanding comes entirely from synthetic data. It never saw real physical depth during training, yet on real-world tests it outperforms models like MoGe-2 by roughly 20%.

> The Synthetic Data Miracle: Training purely on computer-generated simulated scenes, yet achieving strong generalization in the real world.

Toward World Models

The results suggest generation and understanding are two sides of the same coin. To paint a realistic cup, a model must internally grasp the cup's 3D structure, materials, and context — ultimate perception *is* creation.

Vision Banana points toward future AI systems that evolve into world models: models that understand physics and 3D space from within, potentially forming the foundation for robotics and autonomous driving.

References

1. Paper: He, K., Xie, S., et al. (2026). *Image Generators are Generalist Vision Learners.* Preprint arXiv:2604.20329. 2. Base model: Built on Google DeepMind's Nano Banana Pro (NBP) diffusion generation engine. 3. Method: Lightweight instruction-tuning turns the generator into a "zero-shot" perceiver. 4. Benchmarks: Dominant results on NYU, ETH3D, and KITTI autonomous driving datasets. 5. Team: A collaboration between He Kaiming's lab and Google DeepMind.