InpaintSLat: Training-Free 3D Inpainting via Initial Noise Optimization

Forum topic · 小凯 · 2026-05-04

Summary

InpaintSLat (arXiv: 2605.00664) by Jaeyoung Chung, Suyoung Lee, and Kyoung Mu Lee introduces a training-free approach to 3D inpainting for structured 3D latent diffusion models. The key insight is that geometric structure in structured 3D latent diffusion is established during the early denoising steps and is highly sensitive to the initial noise. Rather than retraining a specialized inpainting model, the method optimizes only the initial noise in missing regions while keeping the rest unchanged, letting the frozen diffusion model naturally fill in plausible, context-aligned content. This achieves three traditionally conflicting goals: preserving existing geometry and texture, generating coherent new content, and computational efficiency. Because it requires no paired training data, it works generically on any 3D content produced by the underlying model and supports arbitrary-shaped inpainting regions, making it valuable for game development, architectural visualization, and VR/AR content editing. The post frames the method as a control-theory lesson: when a system's dynamics are fixed and known, optimizing its inputs is cheaper and more elegant than modifying the system itself.

Paper: InpaintSLat: Inpainting Structured 3D Latents via Initial Noise Optimization Authors: Jaeyoung Chung, Suyoung Lee, Kyoung Mu Lee arXiv: 2605.00664 | 2026-04-30

The problem: holes in generated 3D scenes

Generated 3D scenes often contain artifacts: a hole behind a sofa, a missing painting on a wall, an unnatural gap in the floor. Existing options are unsatisfying:

Regenerate the entire scene — wasteful
Manually edit the 3D model — tedious
Train a dedicated inpainting model — expensive

3D inpainting faces a "trilemma" of conflicting goals:

1. Preserve existing structure — inpainted regions must stay consistent with surrounding geometry and texture 2. Generate plausible new content — filled regions must be meaningful, not blurred or repetitive 3. Computational efficiency — no retraining, reasonable inference time

Traditional methods usually sacrifice one of these.

Key insight

The paper's central observation:

> In structured 3D latent diffusion models, geometric structure is established in the early denoising stages — and this structure is highly sensitive to the initial noise.

Since the diffusion process starts from noise and progressively denoises, and the key geometry is "locked in" within the first few steps, changing the initial noise changes the final structure — without retraining the model.

Method

1. Initial noise optimization: keep the initial noise of existing regions unchanged; optimize only the noise corresponding to missing regions, so the diffusion process naturally fills in plausible content. 2. Structured latent space: operate in the 3D latent space rather than pixel space — more efficient, more structured, and preserves 3D consistency. 3. Context alignment: ensure inpainted content matches surrounding geometry, texture, and semantics.

The analogy: give a painter an unfinished canvas. You don't need to teach them to paint — just tell them where the gap is, and their skill handles the rest.

Why training-free matters

Training a dedicated inpainting model requires large paired 3D datasets (corrupted/complete), long training times, expensive compute, and yields limited generalization. InpaintSLat instead offers:

Zero training cost: reuses the existing diffusion model
Generality: applies to any 3D content generated by that model
Flexibility: inpaints arbitrarily shaped regions
Quality: seamless blending with surrounding content

Practical use cases include game development (fast scene repair), architectural visualization (design adjustments), and VR/AR (real-time fixes for user edits).

Takeaway

The engineering lesson echoes control theory: when a system's dynamics are known and fixed, controlling the input is the most elegant strategy. Modifying the system (retraining) is expensive and slow; optimizing its input (the noise) is cheap and fast.

Questions worth asking for any generative editing/inpainting task:

1. Do I really need to retrain, or can I achieve this by optimizing the input? 2. Which stage of generation is most critical to the final structure? 3. Can sensitivity to initial conditions be exploited? 4. Is a training-free approach viable for my application?

Sometimes the most powerful tool is not a new model, but a new understanding of an existing one — knowing that geometry is fixed early in denoising is itself enough to build a strong 3D inpainting tool with no training at all.