Instant NuRec: NVIDIA's Feed-Forward 3D Gaussian Reconstruction Cuts Driving Scene Simulation from 75 Minutes to 1.5 Seconds

Forum topic · QianXun · 2026-10-02

Summary

NVIDIA's Spatial Intelligence Lab introduced Instant NuRec (arXiv:2607.14203v1, July 2026), a feed-forward model that reconstructs drivable neural simulation worlds from 10-20 second multi-camera driving clips in about 1.5 seconds, replacing per-scene optimization that traditionally takes roughly 75 minutes per log. The model outputs a hierarchical 3D Gaussian Splatting scene with four layers: static 3DGS, dynamic 3DGS with three-keyframe trajectories, a cubemap sky, and per-camera ISP correction, exportable to USDZ for closed-loop simulators like NuRec and AlpaSim. Its architecture reuses a DINOv2 ViT with alternating intra-image/cross-image attention and five decoder heads (depth, semantics, motion, sky, 3DGS attributes). On the Waymo Open Dataset it reaches 28.26 dB PSNR, beating the strongest baseline DGGT by 2.01 dB, with larger gains in dynamic regions, and reduces compute by 10^3-10^4x versus per-scene optimization. Critically, closed-loop policy rankings under Instant NuRec match those under the expensive optimized NuRec. A LiDAR branch reconstructs sweeps in about 20 seconds, 225x faster, though geometry coverage is weaker. Stated limitations include undersampling thin structures, limited rig generalization, and inability to express sub-second non-rigid motion.

Autonomous-vehicle closed-loop simulation has been dominated by per-scene optimization methods like NuRec (with OmniRe as a representative), which faithfully reconstruct a driving log but take hours of compute plus careful hyperparameter tuning and auxiliary inputs (LiDAR sweeps, fitted poses, semantic masks). With fleets collecting millions of clips daily, that cost structure cannot scale.

On July 15, 2026, NVIDIA's Spatial Intelligence Lab released the opposite answer: one forward pass, 1.5 seconds.

Paper: *Instant NuRec: Feed-Forward 3D Gaussian Reconstruction for Driving Scene Simulation*, arXiv:2607.14203v1, 2026-07-15 (cs.GR, cs.AI, cs.CV)
Project page: research.nvidia.com/labs/sil/projects/instant-nurec; code and models released

The Hierarchical Output

Instant NuRec takes 10-20 second clips from 1-5 calibrated cameras and outputs an editable, re-timable hierarchical 3D Gaussian world:

| Layer | Content | Attributes | |---|---|---| | Static 3DGS | Road, buildings, rigid background | position, scale, rotation quaternion, opacity, color, normals, semantics | | Dynamic 3DGS | Vehicles and vulnerable road users | same, with piecewise-linear three-keyframe trajectories | | Sky | Six-face cubemap | high-frequency cloud/sun texture | | Per-camera ISP correction | 3x4 RGB affine | initialized to identity; compensates white balance, gamma, vignetting |

Results export to USDZ and feed directly into NuRec and AlpaSim. Rendering uses 3DGUT, natively supporting non-pinhole camera models.

Architecture

A ViT encoder reusing DINOv2 alternates intra-image and cross-image attention; images are split into 14x14 patches plus a per-image class token generated from sinusoidal encodings of the camera's 6-DoF pose and FOV. A DPT head fuses multi-layer features. Five decoder heads predict: dense depth; four-class semantics (road, movable, sky, ego-car); 6-DoF displacement lifted from query points to build three trajectory points; cubemap sky via ray-direction cross-attention; and 3DGS attributes (scale, rotation, opacity, plus color, normals, semantics). A normal head and ISP decode head round out the design.

Benchmarks on Waymo Open Dataset

| Method | PSNR | SSIM | AbsRel | δ₁ | Dynamic PSNR | Dynamic δ₁ | |---|---|---|---|---|---|---| | DepthSplat | 22.48 | 0.645 | 0.295 | 0.592 | 18.59 | 0.512 | | STORM | 21.88 | 0.752 | 0.123 | 0.870 | 19.89 | 0.789 | | Depth-Anything-3 | 20.30 | 0.557 | 0.434 | 0.313 | 17.59 | 0.354 | | DGGT | 26.25 | 0.805 | 0.135 | 0.841 | 21.76 | 0.689 | | Instant NuRec | 28.26 | 0.859 | 0.076 | 0.937 | 24.93 | 0.922 |

+2.01 dB over the strongest baseline DGGT, with larger gains in dynamic regions (24.93 vs 21.76)
Protocol: 2-second windows, 20 consecutive frames, 4 context frames at 0.5 s spacing, RGB resized to 240x160
10³–10⁴x compute reduction vs per-scene optimization (per the abstract)

Ablations

On an internal validation set (full model: 29.93 PSNR, 0.793 SSIM, 0.069 AbsRel, 0.950 δ₁):

Removing the depth loss barely hurts appearance (28.92) but drops δ₁ from 0.950 to 0.862 — the model overfits looks while geometry collapses
Removing frustum-ownership merging is the harshest hit (26.10 PSNR); this module assigns each Gaussian to the nearest-observed chunk to reduce duplicates and floaters in long clips
Replacing the cubemap sky with an MLP costs 3.2 dB

Closed-Loop Rankings

Evaluated in AlpaSim's closed-loop stack (140 scenes, 20 s rollouts, replanning every 500 ms, 6 trials each, five policies from Alpamayo 1.5 variants, VaVAM, and Alpamayo R1):

Core result: policy rankings under Instant NuRec match rankings under expensive per-scene-optimized NuRec exactly
Reconstruction: NuRec ~75 minutes vs Instant NuRec ~1.5 s under both Dense and Selective query strategies; Gaussian budgets drop ~3x (351K vs 120K per context view)

The exact collision/off-route rates per policy appear only in the paper's Figure 8 bar chart and were not transcribed in the HTML version; cite from the paper PDF.

LiDAR Branch

A lightweight LiDAR branch encodes each sweep as a range map (per-beam range and intensity), decoding M Gaussians per valid ray near measured positions with bounded 3D offsets. A Mip-Splatting-inspired anisotropic scale floor ensures footprints are at least as large as beam spacing:

| Method | Chamfer | Precision | Coverage | Intensity MAE | Time | |---|---|---|---|---|---| | NuRec | 0.204 | 0.080 | 0.124 | 0.028 | ~75 min | | Instant NuRec LiDAR | 0.286 | 0.113 | 0.173 | 0.027 | ~20 s | | Without scale floor | 0.936 | 0.195 | 0.741 | 0.036 | — |

~225x faster; geometry is weaker due to incomplete coverage, not mislocalization. Without the scale floor, Gaussians collapse toward observation rays.

Stated Limitations

Inherent tension between Gaussian count and quality; thin structures like poles and traffic lights are undersampled at small budgets
Generalization limited by training rigs; very low mounting heights or fisheye-only configurations need extra fine-tuning
Three-keyframe linear trajectories cannot express sub-second non-rigid motion (e.g., pedestrian joint articulation)
Long logs are chunked; conditioning the forward pass on prior reconstructions is a proposed evolution
Training: three stages, ~6 days on 8 nodes, batch 1-2 clips per GPU; encoder frozen after training
Internal evaluation supports up to 30-second clips

Why It Matters

Reframing reconstruction from an optimization problem into a forward pass changes the cost structure more than any single metric: a 75-minute-per-clip pipeline can never cover the millions of clips fleets collect daily. Ranking fidelity is what closed-loop simulation needs — and that ranking is preserved. The scale-floor ablation (0.936 vs 0.286 Chamfer) is a transferable lesson: neural reconstruction Gaussians need a geometric floor or they collapse onto rays. Single-camera (V=1) operation works, though thin-structure undersampling under the Selective strategy remains the most practical shortcoming.

Sources: arXiv:2607.14203v1 HTML full text (Tables 1-3, Figures 1-8 and corresponding sections) and the NVIDIA project page research.nvidia.com/labs/sil/projects/instant-nurec.