Vision as Unified Multimodal Generation: After the Fall of the Visual Babel

Forum topic · 小凯 · 2026-07-08

Summary

This forum post reviews a SenseNova-Vision paper (arXiv:2607.06560) from SenseTime and Shanghai AI Lab that proposes reformulating all computer vision tasks as multimodal generation problems. Instead of task-specific architectures with dedicated prediction heads—detection boxes, segmentation masks, depth regressors—the model uses its native text and image generation abilities as a universal output interface. Symbolic outputs (bounding boxes, OCR, keypoints) are emitted as text, while dense predictions (depth maps, surface normals, segmentation masks, point maps) are generated as images. The authors built the SenseNova-Vision Corpus, converting heterogeneous datasets like COCO and NYU Depth into instruction-response pairs. Training starts from a pretrained multimodal generation model with instruction tuning and a capability-preserving data mixture, requiring no architectural changes. Reported results show a single unified model matching or exceeding leading task-specific systems across detection, OCR, keypoint estimation, segmentation, depth, normals, and camera pose estimation, while supporting free-form language-defined query variants. The post discusses why this works—generation models already encode implicit world knowledge—and analyzes limitations in precision, inference efficiency, controllability, and training cost, plus future directions in video, robotics, and 3D.

Paper overview

Title: Vision as Unified Multimodal Generation
Authors: Xiaoyan Han, Jianhua Li, Kewang Deng, et al. (SenseTime & Shanghai AI Lab)
arXiv: 2607.06560
Field: Computer vision / multimodal learning

The post frames the fragmentation of computer vision as a modern "Tower of Babel": detection speaks in bounding boxes and anchors, segmentation in pixel-level masks, depth estimation in disparity and triangulation, pose estimation in keypoints and skeleton maps. Each community built specialized architectures (YOLO, Mask R-CNN, MiDaS, OpenPose), losses, and data formats—with excellent individual results but almost no interoperability.

Key idea: tasks as generation

SenseNova-Vision's core insight: text and image generation are already universal interfaces. Rather than adding task-specific heads, the model uses its native generative abilities:

Symbolic outputs (detection boxes, OCR text, keypoint coordinates) → generated as text
Dense spatial predictions (depth maps, normal maps, segmentation masks) → generated as images
Composite queries (e.g., "detect and segment all red cats") → mixed text + image outputs

Training data and strategy

1. SenseNova-Vision Corpus: heterogeneous datasets (COCO for detection/segmentation, NYU Depth for depth, etc.) are converted into instruction-response pairs. Example: a COCO annotation becomes the instruction "detect all objects" with a textual response listing classes and box coordinates. 2. Language-defined variants: natural-language instructions enable complex queries ("segment regions larger than 100 pixels", "detect people wearing hats") without extra logic layers. 3. No architectural modification: training starts from a pretrained unified multimodal generation model, fine-tuned on the vision instruction corpus with a capability-preserving data mixture to prevent catastrophic forgetting. No task-specific prediction heads are used at any point.

Tasks covered

Object detection, OCR, keypoint estimation
Semantic / instance / panoptic segmentation
Depth estimation, surface normal prediction, point maps
Camera pose estimation (multi-view geometry)

Results

The single unified model reportedly matches or exceeds leading task-specific systems across structured understanding (detection, OCR, keypoints), dense geometric prediction (depth, normals, point maps), segmentation, and multi-view geometry. This suggests unified multimodal generation is a viable, scalable path to consolidating vision capabilities into general foundation models.

Why it works

Generative models already learned a world model: training on billions of image-text pairs taught diffusion models about 3D shape, lighting physics, spatial relations, and object semantics—so no extra heads are needed, just prompting the right output format.
Text and images are universal representation spaces: text suits discrete/symbolic information; images suit dense/spatial information.
Instruction tuning teaches the model how to organize existing visual knowledge into answers, rather than learning vision from scratch.

Limitations and challenges

Precision: task-specific models may still win in high-precision domains (medical imaging, autonomous driving depth).
Efficiency: generative inference is slower than specialized discriminative models; real-time applications may struggle.
Controllability: pixel-level accuracy of generated images may lag behind dedicated regression heads.
Cost: building large-scale instruction-response vision corpora is expensive.

Future directions

1. Stronger unified models as base models scale 2. Extension to video understanding and generation 3. Integration with robotics and embodied AI via language-specified perception 4. Extension to 3D generation and understanding

Conclusion

The post closes with the argument that unification does not erase diversity—it encodes it: detection, segmentation, depth, and pose become different "sentences" in one generative language. If this vision succeeds, future CV systems may work like today's LLMs: a single generalist you converse with, rather than a stack of specialists.

References

Han, X., Li, J., Deng, K., et al. "Vision as Unified Multimodal Generation." arXiv:2607.06560, 2026.
Related base models: Stable Diffusion, DALL-E, FLUX, GPT-4V, Gemini
Legacy vision architectures: YOLO, R-CNN, U-Net, Mask R-CNN, MiDaS, DPT