Gaussian Splatting Meets Video Generation: A Cross-Field Survey of Explicit Representations

Forum topic · ✨步子哥 · 2026-08-25

Summary

This survey examines the convergence of 3D Gaussian Splatting (3DGS) and video generation, organized around three intersection points: (1) dynamic/4D reconstruction, including 4D-GS, Deformable 3D Gaussians, Gaussian-Flow (~125 FPS, 5x faster than per-frame methods), and Scaffold-GS; (2) generative 4D, from DreamGaussian (3D in ~2 minutes) to 2025-2026 video-to-4D systems like GenMOJO, Ground4D (mPSNR 19.43 on DyCheck), and World from Motion (DyCheck 19.78 mPSNR), which use diffusion priors to shape Gaussian geometry; and (3) GS as a canvas for video and world models, exemplified by GaussianVideo (ICCV 2025, hierarchical GS + Neural ODE), WorldWarp (200-frame geometrically consistent generation), and D2GV (>400 FPS decoding). Compression work (LightGaussian 15x, 3DGS.zip, Splatwizard) addresses deployment. The analysis weighs strengths (explicitness, real-time 100-300 FPS rendering, editability, data efficiency) against limitations (storage growth, poor topology change handling, photometric-optimization-driven structural drift, inconsistent multi-view diffusion priors). Verdict: the most practical current solution is a conditional hybrid representation-Gaussian geometry as the skeleton, diffusion as the flesh, physics/depth constraints as backfill-pending a unified 4D foundation model.

Gaussian Splatting Meets Video Generation: Explicit Representations Drive the Convergence

*Deep Research cross-cutting survey, APA 7.0 style, completed 2026-08-25.*

Key points

3D Gaussian Splatting (3DGS) represents scenes with millions of learnable Gaussian primitives (position μ, anisotropic covariance Σ, opacity α, spherical-harmonics color), rendered via differentiable rasterization at 100–300 FPS with minutes-scale training (Kerbl et al., 2023). The survey argues that 3DGS and video generation converge because of a shared pain point — consistency: video diffusion models lack geometric constraints and drift across views/time, while 3DGS lacks generative priors for unseen regions. The confluence unfolds along three axes:

1. Dynamic and 4D representation (reconstruction)

| Method | Venue | Mechanism | Key numbers | Limitation | |---|---|---|---|---| | 4D-GS (Wu et al.) | CVPR 2024 | Spatio-temporal encoding + MLP deformation | High-fidelity dynamic rendering | Needs dense multi-view input | | Deformable 3D Gaussians (Yang et al.) | CVPR 2024 | Canonical space + deformation MLP | Monocular dynamic humans | Weak on drastic non-rigid motion | | Gaussian-Flow (Lin et al.) | CVPR 2024 Highlight | Dual-domain deformation (polynomial + Fourier) | 5x faster training, ~125 FPS | Long-range violent motion limited | | Scaffold-GS (Lu et al.) | CVPR 2024 | Anchors + neural Gaussians, view-adaptive | Less redundancy, higher quality | Limited dynamic extension | | MoDec-GS / MoSca / OriGS / ODE-GS / EvoGS | CVPR/NeurIPS 2025 | Motion decomposition, 4D scaffolds, ODE dynamics | Temporal extrapolation | Fragmented, not unified |

Critique: these methods excel only in observed domains; under occlusion, topology change, or viewpoint extrapolation, purely photometric deformation fields exhibit structural drift (empirically shown by Ground4D).

2. Generative 4D (creation)

Miao et al. (2025) categorize 4D generation into four paradigms: end-to-end, intermediate-data generation (multi-view/multi-time video → 4D reconstruction), Score Distillation Sampling (SDS), and explicit supervision.

Text/image → 4D: DreamGaussian (ICLR 2024) creates 3D in ~2 minutes (~10x speedup); GaussianDreamer bridges 2D and 3D diffusion.
Video → 4D (2025–2026 frontier):
GenMOJO (CMU+TRI, 2025): multi-object occluded scenes from monocular video.
Ground4D (2026, arXiv:2606.28828): VGGT-based geometry-free initialization + dynamic GS refinement; 19.43 dB mPSNR and Chamfer 0.226 on DyCheck.
World from Motion / WfM (Stanford+NVIDIA, 2026, arXiv:2607.01202): a closed loop — reconstruct dynamic 3DGS, render RGB/depth/normal/3D-scene-flow buffers to condition a video generation model into "virtual multi-views," then back-project depth, densify, and re-optimize motion. SOTA on DyCheck (19.78 mPSNR) and MultiCamVideo (27.43 PSNR).

3. GS as a canvas for video and world models

GaussianVideo (Koc+Adobe, ICCV 2025, arXiv:2501.04782): hierarchical GS + Neural ODE camera motion + B-spline motion; no pre-computed camera parameters; semantically coherent Gaussian motion emerges naturally.
WorldWarp (2025, arXiv:2512.19678): online 3DGS geometry cache as structural anchor + spatio-temporal diffusion (ST-Diff) with time-varying noise schedules; generates 200-frame long-range geometrically consistent video chunk by chunk.
D2GV (2025, arXiv:2503.05600): deformable 2D Gaussians for video, GoP-parallel CUDA rasterization, >400 FPS decoding, quality comparable to INR; supports interpolation, inpainting, denoising.

Compression and efficiency

LightGaussian (NeurIPS 2024 Spotlight, arXiv:2311.17245): 15x compression, 200+ FPS.
3DGS.zip (CGF 2025): unified survey of nine compression module families.
Splatwizard (2025, arXiv:2512.24742): benchmark toolkit making compression measurable.

Pro vs. con: does GS really advance video generation?

Pro (technical optimism): 1. Explicit = controllable: per-primitive editing, rigging; superior to NeRF black boxes and raw pixels. 2. Real-time = deployable: 100+ FPS enables interactive worlds, digital humans, driving simulation. 3. Composable = systems: GS as canvas to which diffusion, physics, and trajectories attach (WfM, WorldWarp). 4. Data-efficient: strong-modality priors shape weak-modality tasks, avoiding end-to-end big-data demands.

Con (limitations): 1. Storage/VRAM grows linearly with scene complexity. 2. Topology changes, large non-rigid deformation, smoke/hair remain hard (TagSplat, CVPR 2026, flags this as open). 3. Photometric-only optimization causes structural and long-range temporal drift (Ground4D, WfM). 4. Diffusion priors are "per-frame plausible but not one world" — need geometric back-projection constraints. 5. Transparent/reflective materials render worse than NeRF's volume rendering; aliasing persists.

Verdict

GS does not replace diffusion — it scaffolds it. The best current solution is a conditional hybrid representation: geometry (GS / 3D foundation models) as the skeleton, diffusion (video / multi-view) as the flesh, with physics, trajectories, and depth as backfilled constraints. A unified end-to-end 4D foundation model remains the endgame, but data, unified generation frameworks, and efficiency are not yet in place.

Roadmap

Static 3DGS (Kerbl '23) → dynamic/4D representations (CVPR '24 wave) → generative 4D (2025–26: WfM, Ground4D) → GS as video/world-model canvas (GaussianVideo, WorldWarp, D2GV) → lightweight deployment via compression (LightGaussian, 3DGS.zip, Splatwizard).

Open challenges and benchmarks

Topology-consistent dynamic modeling and cross-view/long-range temporal drift remain open.
Physical plausibility (e.g., PhysTwin, ICCV 2025) is not yet universal.
Benchmarks over-rely on PSNR/SSIM/LPIPS/FVD/CLIP and miss cross-view consistency, long-range drift, and geometry-appearance mismatch failures (Miao et al., 2025).
Current systems split roles (image diffusion for appearance, video diffusion for motion, multi-view for viewpoint) and defer consistency to post-processing.
Large-scale, multi-condition 4D data is scarce, blocking end-to-end foundation models.
Common metrics: PSNR, SSIM, LPIPS, FPS, memory, Chamfer Distance, mPSNR (DyCheck), PCK@0.05.

References

Bagdasarian, M., et al. (2025). *3DGS.zip: A survey of 3D Gaussian Splatting compression*. Computer Graphics Forum.
Bond, A., et al. (2025). GaussianVideo: Efficient video representation via hierarchical Gaussian splatting. *ICCV*. arXiv:2501.04782
Chu, et al. (2025). *GenMOJO: Generating multi-object 4D scenes from monocular video*. arXiv:2506.12716
Fan, Z., et al. (2024). LightGaussian: Unbounded 3D Gaussian compression with 15× reduction and 200+ FPS. *NeurIPS*. arXiv:2311.17245
Guo, H., et al. (2026). TagSplat: Topology-aware Gaussian splatting for dynamic mesh modeling and tracking. *CVPR*. arXiv:2512.01329
Kerbl, B., et al. (2023). 3D Gaussian Splatting for real-time radiance field rendering. *ACM TOG*, 42(4), 139. arXiv:2308.04079
Kong, H., et al. (2025). WorldWarp: Propagating 3D geometry with asynchronous video diffusion. arXiv:2512.19678
Lin, Y., et al. (2024). Gaussian-Flow: 4D reconstruction with dynamic 3D Gaussian particle. *CVPR*, 21136–21145. arXiv:2312.03431
Liu, M., et al. (2025). D2GV: Deformable 2D Gaussian splatting for video representation in 400FPS. arXiv:2503.05600
Liu, et al. (2025). *Splatwizard: A toolkit for benchmarking 3D Gaussian splatting compression*. arXiv:2512.24742
Lu, T., et al. (2024). Scaffold-GS: Structured 3D Gaussians for view-adaptive rendering. *CVPR*. arXiv:2312.00109
Miao, Q., et al. (2025). Advances in 4D generation: A survey. arXiv:2503.14501
Tang, J., et al. (2024). DreamGaussian: Generative Gaussian splatting for efficient 3D content creation. *ICLR*. arXiv:2309.16653
Wu, G., et al. (2024). 4D Gaussian Splatting for real-time dynamic scene rendering. *CVPR*. arXiv:2310.08528
Yang, Z., et al. (2024). Deformable 3D Gaussians for animatable humans from monocular videos. *CVPR*. arXiv:2309.13101
Yi, T., et al. (2023). GaussianDreamer: Text to manipulable 3D Gaussian objects with generated multi-view images. arXiv:2310.03585
Zhao, M., et al. (2025). Advances in 4D representation: Geometry, motion and interaction. *ICLR*.
Zhao, Q., et al. (2026). Ground4D: Consistency-aware 4D reconstruction from monocular video. arXiv:2606.28828
Zhu, L., et al. (2026). World from Motion: Generative dynamic Gaussian reconstruction from monocular video. arXiv:2607.01202