Gaussian Splatting Meets Video Generation: How Explicit Representations Are Driving a Convergence

*Deep research cross-cutting survey · APA 7.0 style · August 25, 2026*

Abstract

3D Gaussian Splatting (3DGS) pushed neural rendering into the real-time era with "explicit primitives + differentiable rasterization." But as an inherently static representation, why does it intersect with video generation? This survey identifies three convergence points: dynamic and 4D representations (reconstruction), generative 4D (diffusion priors sculpting shape), and GS as a canvas for video and world models. After a pro/con debate, the conclusion: the current best solution is a conditional hybrid representation—geometry as the skeleton, diffusion as the flesh.

Why the Two Streams Converge

Video generation has traditionally been built on "pixel-grid temporal sequences." Diffusion models excel at appearance continuity but struggle with multi-view and temporal consistency—each generated frame may look plausible, yet frames need not belong to the same world. Meanwhile, 3DGS represents scenes with millions of learnable Gaussian primitives (position μ, covariance Σ, opacity α, spherical harmonics SH), with differentiable rasterization enabling minute-level training and hundreds of FPS rendering (Kerbl et al., 2023).

The two streams converge over a shared pain point: consistency. Video generation drifts without geometric constraints; 3DGS lacks generative priors for unseen regions. Research from 2023–2026 has progressively moved from "each excelling in its own arena" toward "GS as the canvas, diffusion as the brush."

Technical Foundation: What Is 3DGS

3DGS replaces NeRF's implicit volume fields with explicit, differentiable, editable Gaussian primitives. Each primitive has: position μ∈ℝ³, anisotropic covariance Σ=R S Sᵀ Rᵀ (rotation R, scale S), opacity α, and view-dependent color (spherical harmonics). Rendering projects each primitive to a 2D elliptical "splat," alpha-blended in depth order with backpropagation.

Three strengths: training in minutes (vs. hours for NeRF), 100–300 FPS rendering, and explicit primitives enabling pruning, editing, and densification. Weaknesses follow from the same design: primitive counts grow linearly with scene complexity (storage/memory bloat), and the static framework says nothing about time.

Convergence Point 1: Dynamic and 4D Representation (Reconstruction)

| Method | Venue | Mechanism | Key Numbers | Limitation | |---|---|---|---|---| | 4D-GS (Wu et al.) | CVPR 2024 | Spatio-temporal structural encoding + MLP deformation decoding | High-fidelity dynamic reconstruction | Relies on dense multi-view input | | Deformable 3D Gaussians (Yang et al.) | CVPR 2024 | Canonical space + deformation MLP | Monocular dynamic humans | Weak on extreme non-rigid deformation | | Gaussian-Flow (Lin et al.) | CVPR 2024 Highlight | Dual-domain deformation (DDDM): polynomial in time + Fourier in frequency | ~5× faster than per-frame training, ~125 FPS | Limited for long-range, violent motion | | Scaffold-GS (Lu et al.) | CVPR 2024 | Anchor points + neural Gaussians, view-adaptive | Reduced redundancy, higher quality | Limited dynamic extension | | MoDec-GS / MoSca / OriGS / ODE-GS / EvoGS | CVPR/NeurIPS 2025 | Motion decomposition, 4D scaffolds, ODE continuous dynamics | Temporal extrapolation, sample-efficient | Fragmented, not yet unified |

Devil's Advocate #1: These methods win only in "already-observed" domains. Under occlusion, topology changes, or viewpoint extrapolation, purely photometric deformation fields exhibit structural drift (empirically shown by Ground4D). This is not a minor issue—it is the fundamental weakness of dynamic GS.

Convergence Point 2: Generative 4D (Creation)

Reconstruction can only "reproduce what was filmed"; generation can "create from nothing." The difficulty of 4D generation is not an extra time dimension, but keeping four axes stable simultaneously: geometry, appearance, motion, and cross-view consistency. Miao et al. (2025) categorize paradigms into four types:

1. End-to-end: directly learn the 4D distribution (the ideal endpoint, but data-hungry); 2. Generate intermediate data: first produce multi-view/multi-time videos, then feed existing 4D reconstruction pipelines; 3. Implicit distillation (SDS): the diffusion model acts as judge—render, query the difference, backpropagate—sculpting the 4D representation (flexible but expensive); 4. Explicit supervision: directly supervise with reference video/multi-view/geometry (efficient and controllable, provided upstream data is reliable).

Text/Image → 4D: DreamGaussian (Tang et al., ICLR 2024) produced 3D in ~2 minutes (~10× speedup), pioneering "GS + generation"; GaussianDreamer (Yi et al., 2023) bridges 2D and 3D diffusion to create manipulable Gaussian objects.

Video → 4D (the 2026 frontier):

• GenMOJO (Chu et al., CMU+TRI, 2025): video-to-4D in multi-object occluded scenes;
• Ground4D (Zhao et al., 2026, arXiv:2606.28828): training-free geometric initialization via the 3D foundation model VGGT, then refinement with dynamic GS—achieving 19.43 dB mPSNR and Chamfer 0.226 on DyCheck, and exposing the "purely photometric optimization causes structural drift" problem;
• World from Motion (WfM) (Zhu et al., Stanford+NVIDIA, 2026, arXiv:2607.01202): an elegant closed loop—reconstruct a dynamic 3DGS → render RGB/Depth/Normal/3D Scene Flow as a 4D buffer → condition a video generation model to synthesize "virtual multi-views" → depth back-projection, Gaussians densification, motion re-optimization—writing everything back into a single consistent 4DGS. DyCheck 19.78 mPSNR, MultiCamVideo 27.43 PSNR—both SOTA.

Convergence Point 3: GS as the Canvas for Video and World Models

• GaussianVideo (Bond et al., Koc+Adobe, ICCV 2025, arXiv:2501.04782): hierarchical GS + Neural ODE for continuous camera motion + B-spline motion + spatio-temporal hierarchical learning. No precomputed camera parameters needed; memory-efficient with strong temporal consistency. Gaussian motion naturally emerges as semantically coherent, rather than being forced by supervised optical flow.
• WorldWarp (Kong et al., 2025, arXiv:2512.19678): an online 3DGS geometry cache as a structural anchor, refined by spatio-temporal diffusion (ST-Diff); time-varying noise scheduling—full noise for blank regions to generate, half noise for warped regions to repair—produces 200-frame long-range geometrically consistent video chunk-by-chunk. The trick: *3D logic guides structure, diffusion logic perfects texture*.
• D2GV (Liu et al., 2025, arXiv:2503.05600): deformable 2D Gaussians for video representation, with GoP-parallel CUDA rasterization, decoding at over 400 FPS, quality matching or exceeding INR baselines, plus frame interpolation, inpainting, and denoising.

Efficiency and Compression: Making Real-Time Practical

The price of explicit representation is bloat. Three works anchor compression research: LightGaussian (Fan et al., NeurIPS 2024 Spotlight, arXiv:2311.17245) achieves 15× compression at 200+ FPS; 3DGS.zip (Bagdasarian et al., CGF 2025) surveys compression across nine module classes; Splatwizard (Liu et al., 2025, arXiv:2512.24742) provides a benchmarking toolkit making compression measurable.

Devil's Advocate #2: Compression often trades away quality and editability. After 15× compression, do the "editable primitive" advantages survive? This must be judged per task, not universally.

Pro/Con Debate: Can GS Really Propel Video Generation Forward?

Pro (technical optimism):

1. Explicit means controllable: every primitive is readable, editable, and riggable—better than NeRF black boxes or raw video pixels; 2. Real-time means deployable: 100+ FPS rendering enables interactive worlds, digital humans, and autonomous driving simulation; 3. Composable means systemic: GS serves as a canvas onto which diffusion, physics, and trajectories can be layered (WfM, WorldWarp as evidence); 4. Data-efficient: strong-modality priors sculpt weak-modality signals, avoiding the need for massive end-to-end datasets.

Con (critical limits):

1. Storage/memory bloat: primitive counts grow linearly with complexity—long-range scenes suffer; 2. Topology is hard to reshape: topological changes, extreme non-rigid deformation, smoke and hair remain weak for deformation fields (TagSplat, CVPR 2026, explicitly flags this as open); 3. Weak physical and temporal consistency: purely photometric optimization causes structural and long-range temporal drift (confirmed by Ground4D and WfM); 4. Inconsistent multi-view priors: diffusion generates "each frame plausible, together not one world"—geometric back-projection constraints are required; 5. Transparent/reflective materials weak: inferior to NeRF's volume rendering; aliasing and flickering persist.

Verdict (Devil's Advocate integration): GS does not "replace diffusion"—it is the explicit skeleton that "carries" diffusion. The current best solution is a conditional hybrid representation: geometry (GS / 3D foundation models) as the bone, diffusion (video / multi-view) as the flesh, physics/trajectory/depth as constraints filled back in. An end-to-end 4D foundation model is the ultimate destination, but until data, a unified generative framework, and efficiency all mature, hybrid representations are the most practical path. This aligns with Miao et al. (2025)'s methodology: "first find the representation that carries the problem, then talk about generation."

Technical Roadmap and Comparison Matrix

Evolution axis: Static representation (Kerbl'23) → Dynamic 4D (4D-GS, Deformable GS, Gaussian-Flow) → Generative 4D (DreamGaussian → WfM, Ground4D) → Video/world-model canvas (GaussianVideo, WorldWarp, D2GV) → Lightweight deployment (LightGaussian, 3DGS.zip, Splatwizard). Key inflection: 2024 saw the deformation-field paradigm mature; 2025–26 saw generative closed loops and video representations arrive together.

| Method | Venue | Type | Core Mechanism | Speed / Metric | One-line Positioning | |---|---|---|---|---|---| | 3DGS | SIGGRAPH'23 | Static | Explicit Gaussians + differentiable rasterization | 100–300 FPS | Real-time representation foundation | | 4D-GS | CVPR'24 | 4D recon | Spatio-temporal encoding + MLP deformation | High fidelity | Dynamic entry point | | Deformable 3D GS | CVPR'24 | 4D recon | Canonical + deformation field | Monocular human | Deformation-field paradigm | | Gaussian-Flow | CVPR'24 HL | 4D recon | DDDM dual-domain deformation | ~125 FPS, 5× faster | Efficient dynamic | | Scaffold-GS | CVPR'24 | Static/dynamic | Anchor + neural Gaussians | Redundancy reduction | View-adaptive | | DreamGaussian | ICLR'24 | Generative 4D | GS + diffusion | 3D in 2 min | Generation pioneer | | GaussianDreamer | 2023 | Generative 4D | Bridges 2D/3D diffusion | Manipulable | text→4D | | GenMOJO | 2025 | video→4D | Multi-object occlusion reconstruction | DAVIS/MOSE | Occluded scenes | | Ground4D | 2026 | video→4D | VGGT init + GS refinement | mPSNR 19.43 | Geometric consistency | | World from Motion | 2026 | video→4D | Virtual observation distillation into 3DGS | DyCheck 19.78 | Closed-loop SOTA | | GaussianVideo | ICCV'25 | Video representation | Hierarchical GS + Neural ODE | High/low motion both strong | Proto-world-model | | WorldWarp | 2025 | World model | Online 3DGS cache + ST-Diff | 200 frames | Long-range consistency | | D2GV | 2025 | Video representation | Deformable 2D GS | >400 FPS | Ultra-fast decoding | | LightGaussian | NeurIPS'24 | Compression | Pruning + distillation + hashing | 15×, 200+ FPS | Compression benchmark |

Challenges and Benchmarks

• Topology and consistency: topologically consistent dynamic meshes (TagSplat, CVPR 2026) and cross-view / long-range temporal drift remain open problems;
• Physical plausibility: PhysTwin (ICCV 2025) and similar work champion physics-informed reconstruction, but "moving realistically" is not yet universal;
• Benchmark gap: 4D evaluation mostly reuses PSNR/SSIM/LPIPS/FVD/CLIP, which fail to capture the real failure modes—cross-view inconsistency, long-range drift, geometry-appearance mismatch (warning from Miao et al., 2025);
• Unified framework: today's systems still rely on "image diffusion for appearance, video diffusion for motion, multi-view for perspective," with consistency handled largely in post-processing;
• Data: large-scale, multi-condition, multi-dynamics 4D data remains scarce, blocking end-to-end foundation models.

Common metrics: PSNR, SSIM, LPIPS, FPS, memory footprint, Chamfer Distance, mPSNR (DyCheck), PCK@0.05 (3D track).

Conclusion and Outlook

The convergence of 3DGS and video generation is no accident—it stems from the shared anxiety over "consistency." In three years the field has moved from static representation (Kerbl'23) → dynamic 4D (CVPR'24 wave) → generative 4D (WfM, Ground4D in 2025–26) → video/world-model canvases (GaussianVideo, WorldWarp, D2GV), with compression (LightGaussian et al.) paving the way for deployment.

Final verdict: In the short term, conditional hybrid representations are the practical anchor—geometry as skeleton, diffusion as flesh, physics/depth as backfilled constraints. Long term, the field heads toward a unified 4D generative framework with large-scale multi-condition data. For anyone working on interactive worlds, digital humans, autonomous driving simulation, or embodied intelligence, the GS–video-generation convergence is just beginning.

References

• Bagdasarian, M., et al. (2025). *3DGS.zip: A survey of 3D Gaussian Splatting compression*. Computer Graphics Forum.
• Bond, A., Wang, J.-H., Mai, L., Erdem, E., & Erdem, A. (2025). GaussianVideo: Efficient video representation via hierarchical Gaussian splatting. In *ICCV*. arXiv:2501.04782
• Chu, et al. (2025). *GenMOJO: Generating multi-object 4D scenes from monocular video*. arXiv:2506.12716
• Fan, Z., et al. (2024). LightGaussian: Unbounded 3D Gaussian compression with 15× reduction and 200+ FPS. In *NeurIPS*. arXiv:2311.17245
• Guo, H., Weng, D., Su, M., et al. (2026). TagSplat: Topology-aware Gaussian splatting for dynamic mesh modeling and tracking. In *CVPR*. arXiv:2512.01329
• Kerbl, B., Kopanas, G., Leimkühler, B., & Drettakis, G. (2023). 3D Gaussian Splatting for real-time radiance field rendering. *ACM TOG*, 42(4), 139. arXiv:2308.04079
• Kong, H., Yang, X., Zheng, X., & Wang, X. (2025). WorldWarp: Propagating 3D geometry with asynchronous video diffusion. arXiv:2512.19678
• Lin, Y., Dai, Z., Zhu, S., & Yao, Y. (2024). Gaussian-Flow: 4D reconstruction with dynamic 3D Gaussian particle. In *CVPR* (pp. 21136–21145). arXiv:2312.03431
• Liu, M., Yang, Q., Zhao, M., Huang, H., Yang, L., Li, Z., & Xu, Y. (2025). D2GV: Deformable 2D Gaussian splatting for video representation in 400FPS. arXiv:2503.05600
• Liu, et al. (2025). *Splatwizard: A toolkit for benchmarking 3D Gaussian splatting compression*. arXiv:2512.24742
• Lu, T., et al. (2024). Scaffold-GS: Structured 3D Gaussians for view-adaptive rendering. In *CVPR*. arXiv:2312.00109
• Miao, Q., Li, K., Quan, J., et al. (2025). Advances in 4D generation: A survey. arXiv:2503.14501
• Tang, J., et al. (2024). DreamGaussian: Generative Gaussian splatting for efficient 3D content creation. In *ICLR*. arXiv:2309.16653
• Wu, G., Yi, T., Fang, J., et al. (2024). 4D Gaussian Splatting for real-time dynamic scene rendering. In *CVPR*. arXiv:2310.08528
• Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., & Jin, X. (2024). Deformable 3D Gaussians for animatable humans from monocular videos. In *CVPR*. arXiv:2309.13101
• Yi, T., et al. (2023). GaussianDreamer: Text to manipulable 3D Gaussian objects with generated multi-view images. arXiv:2310.03585
• Zhao, M., et al. (2025). Advances in 4D representation: Geometry, motion and interaction. In *ICLR*. (4DRep-GMI)
• Zhao, Q., Deng, W., Wei, P., & Lin, L. (2026). Ground4D: Consistency-aware 4D reconstruction from monocular video. arXiv:2606.28828
• Zhu, L., Huang, S., Mazumdar, A., Li, T., Gojcic, Z., Wetzstein, G., Armeni, I., De Mello, S., & Trevithick, A. (2026). World from Motion: Generative dynamic Gaussian reconstruction from monocular video. arXiv:2607.01202