LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation (arXiv 2610.12442)

Forum topic · 小凯 · 2026-10-10

Summary

LEGO (arXiv 2610.12442) addresses novel view synthesis for generating egocentric video from a single exocentric recording, a hard case where the two cameras share little overlap and much of the target view is unobserved. Existing state-of-the-art pipelines reconstruct the scene explicitly: they estimate depth, lift the video into a point cloud, re-render it from the egocentric camera, and use the result to condition a video diffusion model. This deterministic mapping preserves texture but turns depth errors into misplaced content. LEGO instead proposes a lifting-free approach: a learned view synthesizer, an LVSM-style transformer fine-tuned to render the egocentric view directly without depth, point clouds, or reprojection. Its probabilistic mapping averages over candidate source locations, preserving structure while softening fine texture—ideal for a diffusion generator whose denoising training recovers details. Confidence maps from the distribution mask low-confidence regions and guide early denoising steps. The method outperforms explicit pipelines and generalizes to other datasets without retraining.

Overview

Field: Computer Vision
Authors: Suhwan Cho, Yonwoo Choi, Soongjin Kim, Jicheol Park, Taegyu Lim
Published: 2026-10-08
arXiv: 2610.12442

Abstract

Generating an egocentric video from a single exocentric recording is a challenging case of novel view synthesis, as the two cameras share little overlap and much of the target view is unobserved. Current state-of-the-art methods reconstruct the scene explicitly by estimating depth, lifting the video into a point cloud, and re-rendering it from the egocentric camera to condition a video diffusion model. This deterministic mapping assigns each pixel to a single reprojected location, which preserves texture but translates depth errors into misplaced content.

The authors ask what a video diffusion model should receive as its condition, and propose a lifting-free answer: a learned view synthesizer — an LVSM-style transformer fine-tuned to render the egocentric view directly, without depth, point clouds, or reprojection, resolving cross-view correspondence internally. This probabilistic mapping averages each region over candidate source positions according to a learned correspondence distribution — preserving structure, but blurring fine texture.

The key insight is that this trade-off suits a diffusion generator well: denoising training excels at recovering detail, so an effective condition should prioritize structural alignment over sharpness. The concentration of the learned distribution also yields per-region confidence, used both to mask low-confidence regions and to guide the generator toward high-confidence regions during early denoising steps when layout is forming.

Key Findings

The proposed method consistently outperforms state-of-the-art explicit reconstruction pipelines.
It generalizes to other datasets without retraining.
Division of labor: the synthesizer provides view structure; the diffusion model fills in detail.

---

*Auto-collected on 2026-10-10.*