Vision as Unified Multimodal Generation: SenseNova-Vision Paper Overview

Forum topic · 小凯 · 2026-07-09

Summary

A forum post introduces the paper "Vision as Unified Multimodal Generation" (arXiv 2507.06833) by Xiaoyang Han, Jianhua Li, and Kewang Deng. The work formulates computer vision as unified multimodal generation, expressing heterogeneous visual tasks in the native text and image generation spaces of a unified multimodal model without task-specific architectures. The proposed system, SenseNova-Vision, uses natural-language instructions and optional visual prompts to specify tasks, regions, or views, producing text for symbolic outputs, images for dense spatial predictions, or mixed text-image outputs for compositional tasks. A large-scale instruction-response corpus, the SenseNova-Vision Corpus, was built by converting diverse computer vision annotations. Trained from an off-the-shelf unified multimodal model with no task-specific heads, the model covers detection, OCR, keypoint estimation, segmentation, depth and surface normal prediction, point maps, and camera pose estimation, matching specialized state-of-the-art systems. Models and corpus are publicly released.

Paper Overview

Field: Computer Vision Authors: Xiaoyang Han, Jianhua Li, Kewang Deng Published: 2025-07-09 arXiv: 2507.06833

Summary (Chinese Abstract Translation)

The authors formulate computer vision as unified multimodal generation, in which heterogeneous visual tasks are represented in the native text and image generation spaces of a unified multimodal model — without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify the task, target regions or views, and decoding conventions, and generates responses as:

Text for symbolic outputs
Images for dense spatial predictions
Mixed text-and-image outputs for compositional tasks

To support large-scale training, the authors convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, forming the SenseNova-Vision Corpus — a computer-vision instruction-response corpus spanning text, image, and mixed targets.

Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is primarily trained on this corpus, supplemented with multimodal data as a capability-retention mixture, with no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of visual tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants combining category, color, region, and other visual cues.

Experiments show that a single unified model can match leading task-specific systems in structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest that unified multimodal generation is a scalable path to integrating computer vision capabilities into general-purpose foundation models. The models and corpus have been publicly released.

Original Abstract (Excerpt)

> We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning ...

---

*Auto-collected on 2026-07-09*