[论文] Can Learning to Answer Spatial Questions Improve Novel-View Generation...

论文概要

研究领域: CV 作者: Jahyeok Koo, Hyeonseo Yu, Seungho Jang, Jaewoo Jung, Jiwon Kang 发布时间: 2026-10-14 arXiv: 2610.20610

中文摘要

从有限观测中生成新视角需要同时保持场景的外观和空间结构。尽管近期方法利用了细粒度几何线索，但它们并未显式建模高层语义和空间属性，如物体身份、属性及物体间关系。本文探究3D语言理解能否为新视角生成提供互补监督。我们提出Ask4NV，基于预训练的统一多模态模型，采用混合Transformer（MoT）架构。模型将相机几何信息融入VAE和ViT表示，并通过模态感知的注意力掩码策略，在生成过程中利用理解特征进行联合自注意力。实验表明，该方法在新视角生成中实现了出色的图像质量和几何一致性，同时保持了有竞争力的3D语言理解能力。进一步分析显示，联合训练改善了VAE表示中的跨视图对应关系和3D感知特征空间的相似性。

原文摘要

Generating novel views from limited observations requires preserving both the appearance and spatial structure of a scene. Although recent approaches exploit fine-grained geometric cues, they do not explicitly model high-level semantic and spatial properties such as object identity, attributes, and inter-object relationships. In this work, we investigate whether 3D language understanding can provide complementary supervision for novel-view generation. We introduce Ask4NV, built on a pretrained unified multimodal model with a Mixture-of-Transformers (MoT) architecture. Our model incorporates camera geometry into both VAE and ViT representations and leverages understanding features during generation through joint self-attention with a modality-aware attention masking strategy. Experiments de...

#论文 #arXiv #CV #小凯