[论文] SM4RT: Cascaded Feed-forward Model for 4D Reconstruction

论文概要

研究领域: CV
作者: Pang-Chi Lo, Jhao-Syun Lai, Tsung-Wei Ke
发布时间: 2026-11-05
arXiv: 2611.08436

中文摘要

我们提出 SM4RT，一个用于从 RGB 视频重建动态场景的多任务级联前馈模型。现有的前馈方法通常分别处理几何重建和运动估计——且往往在单一坐标系中预测运动，必须考虑相机运动的影响——而 SM4RT 按顺序解决这些任务：首先重建 3D 几何，然后从预测的结构引导运动估计。通过将每个像素的 3D 运动表达在其对应的相机坐标系中，SM4RT 减少了相机运动带来的歧义。为应对真实世界运动标注的稀疏性，SM4RT 进一步引入了弱监督损失，利用另一时刻的已标注点位置来正则化未标注点的运动。我们在三个动态场景数据集——Kubric、Waymo 和 Point Odyssey——上评估运动估计，SM4RT 在大多数指标上优于已有基线。我们还在五个动态场景数据集上基准测试了几何重建，表明 SM4RT 与最先进方法性能相当。总体而言，我们的统一框架表明，将几何重建和运动估计以级联方式结合不仅避免了性能折衷，还通过改进的几何推理增强了运动估计。

原文摘要

We introduce SM4RT, a multi-task cascade feed-forward model for dynamic scene reconstruction from RGB videos. While existing feed-forward approaches typically address geometry reconstruction and motion estimation separately--and often predict motions in a single coordinate frame that must account for camera movement--SM4RT solves these tasks sequentially by first reconstructing 3D geometry and then bootstrapping motion estimation from the predicted structure. By expressing each pixel's 3D motion in its corresponding camera frame, SM4RT reduces ambiguity arising from camera motion. To address the sparsity of real-world motion labels, SM4RT further incorporates a weakly supervised loss that regularizes unlabeled point motions using labeled point positions at the other time step. We evaluate motion estimation on three dynamic-scene datasets--Kubric, Waymo, and Point Odyssey--and SM4RT achieves the best performance on most metrics compared with prior baselines. We additionally benchmark geometry reconstruction on five dynamic scene datasets and show that SM4RT achieves competitive performance with state-of-the-art methods. Overall, our unified framework demonstrates that combining geometry reconstruction and motion estimation in a cascade not only avoids performance trade-offs but also enhances motion estimation through improved geometric reasoning.

自动采集于 2026-11-08

#论文 #arXiv #CV #小凯