Envisioning the Future, One Step at a Time: How AI Learns to "Foresee"
> "The best way to predict the future is to invent it. But before we can invent it, we must be able to imagine it." — Alan Kay
This post is a Feynman-style deep-dive into the paper "Envisioning the Future, One Step at a Time" by Stefan Andreas Baumann, Jannik Wiese, Tommaso Martorella, et al. from the Technical University of Munich (TUM) and the Max Planck Institute.
The Problem: Predicting One Future vs. Many
Humans constantly simulate the future unconsciously — predicting how waves will break, how a car ahead will slow down, or where a running child will cross. These predictions are inherently uncertain, and it is precisely this modeling of uncertainty that lets us survive in a complex world.
Existing approaches to video future prediction fall into three categories, each with fundamental limitations:
- Deterministic video generation models (VAE/GAN-based): cannot model uncertainty; always produce a single "average" future.
- Probabilistic latent variable models (e.g., SVG, SAVP): can sample multiple futures but tend to produce blurry or unrealistic outputs and are computationally expensive per sample.
- Physics-based simulators (e.g., Bullet, MuJoCo): accurate, but require precise 3D scene reconstruction and physical parameters — nearly impossible from a single 2D photo.
- More tractable than pixels: predicting ~200 coordinates beats predicting 2 million pixel values.
- More physical: point trajectories carry explicit physical meaning (gravity, elasticity, friction).
- Below the object level: unlike objects, points avoid the difficult "binding problem" and can represent amorphous motion like water waves, smoke, or fluttering flags.
- Locality: each point's motion depends mainly on its neighborhood, enabling efficient scaling to thousands of points.
- Uncertainty accumulation: small early differences naturally amplify, mirroring real-world chaos (the butterfly effect).
- Long-horizon consistency: step-by-step generation avoids frame-jumping artifacts.
- Controllability: noise levels can be adjusted to produce conservative or diverse predictions.
- Per-point distributions: each point's future is a Gaussian with a mean (most likely position) and variance that grows over time.
- Cross-point correlations (covariance): points on the same rigid body move coherently, preventing physically impossible futures like an object's halves moving in opposite directions.
- Open-set scenarios from in-the-wild videos (landscapes, streets, interiors, sports), including scenes unseen at training time.
- Long-horizon prediction of up to 10 seconds (~300 frames at 30 fps), 5–10x longer than typical benchmarks.
- Multi-modal evaluation of diversity and uncertainty calibration, with real (not synthetic) uncertainty.
- Accuracy: error reduced by roughly 30% at 1 second, ~50% at 5 seconds, and ~60% at 10 seconds.
- Diversity: thousands of distinct futures, generated orders of magnitude faster (hundreds of samples per second).
- Physical plausibility: no objects vanishing or passing through walls; consistent rigid-body relations.
- Efficiency: ~10x faster forward passes by processing sparse points instead of full pixels.
- Autonomous driving: generating probabilistic futures (e.g., vehicle goes straight 70% / changes lanes 20% / brakes suddenly 10%) enables safer decisions even for low-probability hazards.
- Robot planning: robots can "imagine" many outcomes before acting, combining sample generation with planners like Monte Carlo Tree Search for robust strategies.
- Creative tools: animation in-betweening, real-time game physics, VFX previews (collapsing buildings, smoke dispersion), and design testing (e.g., how a cup tips over).
- Baumann, S. A., Wiese, J., Martorella, T., et al. (2026). Envisioning the Future, One Step at a Time. arXiv preprint.
- Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models.
- Sohl-Dickstein, J., et al. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics.
- Box, G. E. P. (1979). Robustness in the Strategy of Scientific Model Building.
There is also the open-set prediction challenge: real-world AI must handle scenarios never seen during training (an ostrich darting into the road, a falling billboard), requiring a generalized understanding of physics rather than memorized patterns.
Core Insight: The Future Lies in Points, Not Pixels
The paper's key idea: the key to predicting the future is not generating pixels, but predicting the trajectories of points.
Instead of forecasting millions of pixel values, the model tracks a small set of sparse points (e.g., ~100) and predicts where they move. This abstraction is deliberate:
The hierarchy: sparse points → trajectories → trajectory sets → a distribution over future trajectory sets, which encodes uncertainty about the future.
Method: Autoregressive Diffusion
The method applies diffusion models — successful in image generation (DALL-E, Stable Diffusion, Midjourney) — to time-series prediction in an autoregressive manner:
1. Extract sparse points from the current frame at time t. 2. Use the diffusion model to sample multiple possible positions at t+Δt. 3. For each sample, repeat the process for t+2Δt, and so on.
Benefits:
A key technical balance is local prediction with global consistency: each point's motion is predicted locally, while an attention mechanism preserves global coherence when distant events matter.
Uncertainty is modeled at two levels:
The model also supports conditional sampling: constraints such as "the ball lands in the basket" can be injected via conditional diffusion, turning the model into a planning tool that can answer "how should I throw to score?"
The OWM Benchmark and Results
The researchers introduce OWM (Open-World Motion Prediction), designed to fix flaws of existing benchmarks (closed sets, deterministic evaluation, short horizons):
Reported results versus baselines:
Ablations confirm each design choice: dense pixels hurt accuracy and cost more; one-shot generation degrades at long horizons; removing explicit uncertainty collapses outputs to an "average future"; removing global attention hurts complex interaction scenes.
Applications
Conclusion
As statistician George Box said: "All models are wrong, but some are useful." An AI that predicts the future need not be a perfect physics simulator — it only needs to be good enough to help us make better decisions, create better art, and build a safer world. Perhaps the essence of intelligence is:
Not seeing reality, but seeing possibility.