FLAT: Growing a Walkable 3D World from a Single Photo
Paper: FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation Authors: Orest Kupyn, Goutam Bhat, Philipp Henzler, et al. arXiv: 2606.24876 Released: 2026-06-23
This post is a deep-dive explainer (in Feynman style, originally in Chinese) of FLAT, a new scene generation method that converts one photograph into a geometrically accurate 3D scene you can actually walk into.
The Problem: Why "Stepping into a Photo" Is Hard
Generating a 3D scene from a single 2D photo is an ill-posed inverse problem: a photo is a projection of the 3D world that loses depth, backsides, and occluded regions. Humans fill in the gaps using lifelong statistical priors; AI systems need equivalent priors, which modern video diffusion models provide after training on massive video data.
The standard pipeline today is: 1. Generate a multi-view video from the input image with a video diffusion model 2. Reconstruct a 3D representation from the video 3. Render it in real time
The bottleneck is step 2. Most methods use 3D Gaussian Splatting (3DGS) — millions of colored Gaussian blobs. 3DGS renders fast and looks good, but has no explicit surface. You cannot measure exact dimensions, place virtual objects on surfaces, or export the scene into a game engine, because 3DGS is geometrically ambiguous: it "looks right" but isn't truly right.
FLAT's Insight: Triangles Instead of Blobs
FLAT argues that if you want geometrically accurate scenes, you should directly predict a geometrically accurate representation: triangle splats, the fundamental primitive of computer graphics.
Triangles offer:
- Explicit surfaces with well-defined boundaries and normals
- Measurable geometry (distances, angles, areas)
- Compatibility with any standard graphics pipeline (game engines, CAD, renderers)
- Editability (deformation, cutting, merging)
- position along the ray (depth)
- spin around the ray
- tilt relative to the ray
- Removes redundant degrees of freedom: spinning a flat triangle around its own normal changes nothing visually, so only 2 rotational degrees of freedom need to be predicted instead of 3–4.
- Zero rotation naturally means facing the camera, which is the state of most scene surfaces (walls, floors, tabletops), making the learning problem easier.
- Normal estimation error reduced by over 30% versus the best prior methods; depth error by over 25%; clearly better mesh reconstruction than 3DGS-based methods on real-world datasets like ScanNet.
- Rendering quality (PSNR/SSIM) remains competitive with the best 3DGS approaches.
- Qualitatively: cleaner flat surfaces, sharper edges (no "fuzzy" Gaussian boundaries), and sensible topology — closed objects actually have closed surfaces.
- Ablations confirm each component matters: removing ray-centered parameterization or the product window function hurts; swapping triangles for Gaussians collapses geometric accuracy.
- A controlled representation comparison under identical training settings found: 3DGS looks good but is geometrically poor; 2DGS is slightly better; FLAT's triangle splats are the most geometrically accurate while staying visually competitive.
- Game development: directly import photos as explorable scenes
- Architectural visualization: photo-to-model rapid prototyping
- VR: let users step inside any photo
- Robot simulation: realistic environments for training
- Cultural heritage: old photos turned into explorable virtual museums
- From seeing to understanding: FLAT extracts real-world geometric structure, a step from perception toward cognition.
- Classical representations reborn: the humble triangle, the oldest 3D primitive, gains new life through modern AI.
- From optimization to feedforward: unlike NeRF/3DGS per-scene optimization (minutes to hours), FLAT outputs results in seconds, enabling real-time and interactive deployment.
- Quality is bounded by the underlying video diffusion model (hallucinations propagate).
- Very complex scenes (dense forests, crowded streets) may need finer-grained or hybrid representations (triangles + Gaussians + neural fields).
- Static scenes only; dynamic scenes (moving people, swaying leaves) remain open.
- Global metric scale is still imprecise; SLAM or depth sensors could help.
- Kupyn, O., Bhat, G., Henzler, P., et al. "FLAT: Feedforward Latent Triangle Splatting for Geometrically Accurate Scene Generation." arXiv:2606.24876, 2026.
- Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering," 2023.
- "Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets."
- "NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction."
Why Hasn't Anyone Done This Before?
Triangles are extremely orientation-sensitive. A Gaussian blob looks roughly the same from any angle, but a triangle seen edge-on becomes a thin line, and seen from behind, may be invisible. Tiny orientation errors cause huge visual differences, leading to poor gradient flow during training. The paper calls the task "notoriously more challenging."
Two Key Technical Breakthroughs
1. Ray-Centered Rotation Parameterization
Instead of predicting triangle orientations in a global frame (Euler angles or quaternions), FLAT expresses each triangle's rotation relative to the camera ray passing through its pixel:
Benefits:
It's the difference between telling a robot "point toward global coordinates (40, 116)" versus "look right 10 degrees, down 5 degrees from where you are."
2. Product Window Function
Differentiable rasterization softens hard in/out tests into continuous probabilities. But for small triangles (far away or viewed edge-on), gradients become very weak. FLAT multiplies two window functions — one in the triangle's plane, one along the depth axis — which enlarges the effective gradient region in both directions while preserving geometric precision.
Like two flashlights crossed in fog instead of one: the intersection area is much better lit.
Architecture: Video Latents to Triangles
1. Video diffusion: a model like Stable Video Diffusion generates multi-view latent features from the input image. 2. Latent-to-triangle decoder: a lightweight feedforward network directly decodes the diffusion latents into triangle parameters (position, orientation, size, color) in one forward pass — no per-scene optimization, no per-frame processing. 3. Differentiable rendering: triangles are rendered differentiably, enabling end-to-end training from rendering error. 4. Optional test-time refinement: a few seconds of iterative optimization turns the raw "triangle soup" into clean, closed, game-engine-ready meshes.
Results
Why It Matters: Game-Engine-Ready 3D Assets
3DGS cannot be imported into Unity/Unreal, cannot support physics simulation, collision detection, or precise measurement. FLAT's triangle output is natively compatible with all standard graphics pipelines, enabling:
The test-time refinement also yields watertight, manifold meshes — usable for 3D printing.
Broader Significance
Limitations and Future Work
Future directions: hybrid representations, dynamic triangle deformation, semantic understanding (materials, object categories), and interactive editing of reconstructed scenes.