🔮 One Thousand Futures — How AI Learns to "Foresee"
> "The best way to predict the future is to invent it. But before inventing it, we must first be able to imagine it." — Alan Kay
This is a Feynman-style deep-dive explainer of the paper "Envisioning the Future, One Step at a Time" (Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella, et al.; Technical University of Munich & Max Planck Institute; cs.CV / cs.AI / cs.LG). Project page: http://compvis.github.io/myriad
The Problem: Predicting One Future Is Not Enough
Humans constantly simulate the future unconsciously — judging whether a wave will soak our feet, whether a car will brake, where a running child will cross. These predictions are not exact calculations; they are probabilistic simulations of multiple possibilities.
Current AI future-prediction methods fall into three camps, each with fundamental limits:
- Deterministic video models (VAE/GAN-based): always output a single "average" future; cannot represent uncertainty.
- Probabilistic latent variable models (e.g., SVG, SAVP): can sample multiple futures, but quality is often blurry/artifacted and each sample is computationally expensive.
- Physics-based simulators (Bullet, MuJoCo): accurate but require precise 3D scene reconstruction and physical parameters — impractical from a single 2D photo.
- Uncertainty accumulates naturally over time, like the butterfly effect in real physics.
- Long-horizon consistency, since each step conditions on the previous one.
- Controllability via noise levels and conditional diffusion: constraints like "the ball lands in the basket" can guide sampling — turning the model into a planning tool.
- Open-set: diverse in-the-wild videos (nature, streets, indoors, sports), including unseen scene types at test time.
- Long horizon: up to 10 seconds (~300 frames at 30fps), 5–10x longer than typical benchmarks.
- Multi-modal evaluation: accuracy, diversity, and uncertainty calibration.
- Real uncertainty: real-world video, not synthetic simulation.
- Accuracy: ~30% error reduction at 1s, ~50% at 5s, ~60% at 10s versus baselines — the advantage grows with horizon.
- Diversity: thousands of distinct future samples; generation orders of magnitude faster than baselines (hundreds of samples per second).
- Physical plausibility: no objects vanishing or passing through walls; rigid bodies stay rigid; trajectories respect inertia, gravity, collisions.
- Efficiency: ~10x faster inference by processing sparse points instead of pixels.
- Autonomous driving: generate weighted futures for other agents (e.g., 70% keep straight / 20% lane change / 10% sudden braking) and plan safely even against low-probability hazards.
- Robot planning: imagine many futures before acting; combine with Monte Carlo Tree Search for robust policies under uncertainty.
- Creative tools: animation in-betweening with selectable motions, realistic game physics, VFX previews (debris, smoke), and industrial design testing (e.g., how a cup tips over).
- Baumann, S. A., Wiese, J., Martorella, T., et al. (2026). Envisioning the Future, One Step at a Time. arXiv preprint.
- Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models.
- Sohl-Dickstein, J., et al. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics.
- Amodei, D., & Hernandez, D. (2018). AI and Compute.
- Box, G. E. P. (1979). Robustness in the Strategy of Scientific Model Building.
On top of this, most methods are closed-set: they only work on scene types seen in training. Real intelligence requires open-set prediction — handling an ostrich darting into the road, a falling billboard, or a sudden hailstorm.
The Core Insight: Points, Not Pixels
The paper's key idea: predict the trajectories of sparse points rather than generate pixels.
Why points are the right abstraction:
1. Dimensionality: predicting ~100 tracked points (a few hundred values) is far easier than millions of pixels. 2. Physical interpretability: point motion directly reflects forces — gravity, elasticity, friction. 3. Uncertainty modeling: each point's future can be a probability distribution rather than a fixed position. 4. The sweet spot: pixels are too low-level (like tracking air molecules to forecast weather); objects are too high-level (segmentation is hard, and much motion — waves, smoke, flags — has no object boundaries). Points are local, trackable, and composable.
The hierarchy: sparse points → trajectories → trajectory sets → a distribution over trajectory sets encoding future uncertainty.
Autoregressive Diffusion, One Step at a Time
The model uses a diffusion model applied autoregressively over time:
1. Extract sparse points from the current frame. 2. Use the diffusion model to sample multiple possible point positions at t+Δt. 3. For each sample, repeat for the next timestep, up to the desired horizon.
Benefits:
Prediction is mostly local (a point depends on its neighborhood), enabling efficient scaling to thousands of points, while attention mechanisms preserve global consistency for complex interactions.
The OWM Benchmark
The authors introduce OWM (Open-World Motion Prediction) to address shortcomings of existing benchmarks (closed-set scenes, deterministic evaluation, short horizons):
Reported Results
Ablations confirm each design choice: dense pixels hurt accuracy and cost; one-shot generation degrades over long horizons; removing explicit uncertainty collapses to an "average future"; removing global attention hurts complex interactive scenes.
Applications
Closing Thought
The brain doesn't solve Navier–Stokes equations when watching waves — it runs a useful, approximate, high-level simulation. This paper shows AI can do something similar: not by memorizing pixel patterns, but by understanding point trajectories, physical regularity, and the nature of uncertainty — imagining a thousand possible futures instead of one.
As statistician George Box said: "All models are wrong, but some are useful." The essence of intelligence may be:
Not seeing reality, but seeing possibility.