Key points
What 4D Gaussian Splatting is
- A scene representation that uses millions of explicit 3D Gaussians instead of an implicit neural network (NeRF).
- Each Gaussian carries about 15 parameters: position (x, y, z), color (RGB), opacity (alpha), and a 3D covariance matrix (scale + rotation quaternion).
- Adding the time dimension via a learned deformation field, so a single canonical set of Gaussians can morph into different states at different timestamps.
- Free-viewpoint video for sports, concerts, and replays.
- VFX: Framestore used 4D-GS in the Superman film for volumetric performance capture.
- VR/AR and metaverse: real-time exploration of captured scenes.
- Robotics: Splat-Nav achieves real-time 25 Hz pose estimation and path planning on Gaussian maps.
- Digital humans, virtual idols, telepresence.
- Capture requires dozens of synchronized cameras and controlled lighting; not a phone-shot pipeline yet.
- Topological changes (tearing, splitting) and severe topology changes are hard.
- Semantic editing (resize a dog, recolor clothing) requires extra networks.
- Storage of millions of Gaussians reaches hundreds of MB to several GB; compression and streaming remain open.
- 5D: separate illumination so scenes can be relit (day/night, HDR envmaps).
- 6D: factor out material properties (roughness, metalness, IOR) for physically based rendering.
- Coupling with diffusion models to generate 4D Gaussian scenes directly from text.
- Official code:
https://github.com/hustvl/4DGaussians - Project page:
https://guanjunwu.github.io/4dgs/ - Paper:
https://arxiv.org/abs/2310.08528 - WebGPU viewer:
https://github.com/Scthe/gaussian-splatting-webgpu - Hardware: RTX 3090/4090 (24 GB) recommended; RTX 2080 minimum.
- Original Corridor Crew video: https://www.youtube.com/watch?v=X8yRlA7jqEQ
- 4D-GS paper: https://arxiv.org/abs/2310.08528
- 4D-GS project page: https://guanjunwu.github.io/4dgs/
- 3D-GS (SIGGRAPH 2023): https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
- Splat-Nav (robotics): https://chengine.github.io/splatnav/
- AutoDub (dubbing tool used by the author): https://github.com/king33329/audoDub
The pipeline
1. Initialize Gaussians from a Structure-from-Motion (SfM) sparse point cloud. 2. Encode a 4D query (x, y, z, t) using HexPlane, which decomposes the 4D spacetime volume into six 2D feature planes: XY, XZ, YZ, XT, YT, ZT. 3. Predict per-Gaussian deformation (position, rotation, scale offsets) at time t. 4. Rasterize the deformed Gaussians onto the image plane (splatting), sort by depth, and alpha-composite. 5. Optimize via differentiable rendering using L1 + SSIM loss, with adaptive densification: split Gaussians whose gradients are too large, prune low-opacity ones.Why it beats NeRF
| Method | Representation | Training | Rendering | |---|---|---|---| | NeRF | Implicit MLP | 20-48 hours | <1 FPS | | 3D-GS (static) | Explicit Gaussians | 10-20 min | 100+ FPS | | 4D-GS (dynamic) | Deformed Gaussians + HexPlane | 8-30 min | 30-82 FPS |Speed comes from GPU-friendly rasterization instead of per-pixel ray marching. Reported PSNR is on par with or better than NeRF (25-32 dB) at resolutions up to 1352x1014, with roughly 3-4.5 GB VRAM usage.
Applications
Limitations
Future directions
How to try it
Bottom line
4D Gaussian Splatting shifts the field from implicit neural fields to explicit geometric primitives, achieving real-time dynamic scene rendering with modest hardware. It underpins next-generation free-viewpoint video, volumetric VFX, robotics, and is a likely building block for generative, relightable, and eventually text-to-4D worlds.---