After the Fall of the Visual Tower of Babel: When All Eyes Learn One Language
*Forum review of the paper Vision as Unified Multimodal Generation (arXiv:2607.06560), by Xiaoyang Han, Jianhua Li, Kewang Deng, et al. (SenseTime & Shanghai AI Lab), in computer vision / multimodal learning.*
Background: A Fragmented Field
The post opens with the Tower of Babel metaphor: in computer vision, each task community speaks its own "dialect." Object detection uses boxes and anchors (YOLO, R-CNN), segmentation uses pixel masks (U-Net, Mask R-CNN), depth estimation uses disparity (MiDaS, DPT), and pose estimation uses keypoints (OpenPose). Each of the four task families has evolved dedicated architectures, loss functions, data formats, and evaluation standards:
- Structured understanding — detection, OCR, keypoint detection (discrete, symbolic outputs)
- Dense geometric prediction — depth, surface normals, camera pose (continuous fields)
- Segmentation — semantic, instance, panoptic masks
- Multi-view geometry — 3D reconstruction, calibration, stereo matching
- Symbolic outputs (boxes, OCR, keypoints) → generated text
- Dense spatial predictions (depth, normals, segmentation masks) → generated images
- Composite tasks (e.g., "detect and segment the red cats") → mixed text + image output
- Implicit world models: large text-to-image diffusion models already internalize object shapes, lighting physics, spatial relations, and semantics during pretraining — so generating a depth map or mask leverages existing knowledge.
- Text and images as universal representations: text suits discrete symbolic information; images suit dense spatial information.
- Instruction tuning: rather than learning vision from scratch, the model learns to *organize* existing visual knowledge into the requested output format.
- Specialized models may remain superior for extremely high-precision tasks (medical imaging, autonomous driving)
- Generative inference is slower than discriminative models, challenging real-time use
- Pixel-level controllability of generated outputs may lag dedicated regression heads
- Building the instruction–response corpus is computationally and data-expensive
- Han, X., Li, J., Deng, K., et al. "Vision as Unified Multimodal Generation." arXiv:2607.06560, 2026.
- Related foundation models: Stable Diffusion, DALL-E, FLUX, GPT-4V, Gemini
- Legacy vision architectures: YOLO, R-CNN, U-Net, Mask R-CNN, MiDaS, DPT
These specialized systems excel individually but cannot communicate: a detector cannot segment, a depth model cannot detect keypoints. Chaining multiple models incurs heavy integration cost.
The Core Idea: Generation as a Universal Interface
SenseNova-Vision unifies vision not at the architecture level but at the paradigm level: every vision task is reformulated as a generation problem, using only the model's native text and image generation abilities — no task-specific prediction heads:
Because instructions are natural language, the model supports language-defined variants such as "find all red cars" or "segment regions larger than 100 pixels" without extra logic layers.
SenseNova-Vision Corpus
The authors converted heterogeneous datasets (COCO detection, NYU Depth, etc.) into instruction–response pairs: e.g., a detection annotation becomes a natural-language instruction with a text response listing boxes; a depth pair becomes an instruction with the depth map as the image response. The corpus spans detection, OCR, keypoints, segmentation, depth, normals, point maps, and camera pose estimation.
Training
Rather than training from scratch, SenseNova-Vision fine-tunes a pretrained unified multimodal generation model on the vision instruction corpus, mixed with auxiliary multimodal data (a "capability-preserving mixture") to prevent catastrophic forgetting. No architectural modifications are required.
Supported Tasks and Results
One model covers: object detection, OCR, keypoint estimation, semantic/instance/panoptic segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation — plus free-form compositional queries.
Experiments show this single unified model matches or exceeds leading task-specific systems across structured understanding, dense geometric prediction, segmentation, and multi-view geometry, demonstrating that unified multimodal generation is a viable and scalable path.
Why It Works
Implications and Limitations
Paradigm shift: from task-specific models to a general visual assistant; from labeled datasets to instruction data; from architecture design to prompt engineering. Practically, this means lower integration cost, more flexible capabilities, and faster iteration.
Limitations noted in the post:
Future directions: scaling unified models, extending to video (temporal consistency, motion), integration with robotics and embodied AI, and 3D generation/understanding (NeRF, 3D shapes).
Conclusion
The post closes with the observation that unification does not erase diversity — detection, segmentation, depth, and pose become different "sentences" in one shared generative language. Quoting Wittgenstein's *Tractatus* ("The limits of my language mean the limits of my world"), the author suggests that when all vision tasks share one generative language, AI's visual world becomes correspondingly richer and more unified.