DreamFly: When Drones Learn to Remember and Foresee
> *"Humans are unique not because we can fly, but because we can imagine ourselves flying."* — Antonio Damasio
This post is a richly illustrated Chinese-language walkthrough of the paper "DreamFly: Causal Memory and Receding-Horizon Diffusion Planning for Aerial Vision-Language Navigation" (Deng & Xu, arXiv:2608.12308). Below is a structured summary of its key points.
Key points
The problem: aerial VLN is a 3D maze
- Aerial Vision-Language Navigation asks a drone to follow a natural-language instruction (e.g., "find the red-roofed church") using only onboard camera/microphone, in wind, noise, and partial observability.
- Compared to ground navigation, aerial VLN adds a top-down flattened viewpoint, dynamic disturbances (wind, battery, propeller noise), and linguistic ambiguity ("fly over that forest" — which forest? how high?).
- Existing VLA models (RT-2, OpenVLA, Dream-VLA) suffer from three obstacles: 1. Short historical memory — early landmarks in the instruction are forgotten. 2. Short planning horizon — models optimize only the next step. 3. No explicit stopping — models circle the goal or stop early.
- Principle: at any decision time *t*, the model may only use observations strictly earlier than *t* — no "future leakage."
- Implemented via a three-layer structure: 1. Observation encoding: each frame is encoded into a visual feature vector fused with the language instruction. 2. Causal temporal alignment: a masking mechanism (analogous to autoregressive attention, applied along time) blocks all observations with timestamp ≥ t. 3. Dynamic memory augmentation: a learned weighting mechanism emphasizes task-relevant history and fades stale information.
- Effect: unlike conventional memory models whose test success rates drop sharply relative to training (a symptom of "cheating" on future observations), DreamFly learns genuinely causal navigation strategies that transfer to unseen maps.
- One-shot planners (A*, RRT) fail in partially observable aerial settings: the drone cannot see around corners, wind changes, and target descriptions are ambiguous.
- DreamFly uses a diffusion model as the planner — apt because both image generation and navigation planning are searches for a feasible path through high-dimensional space from a start state to a goal state, and diffusion naturally models multimodal uncertainty (many valid routes to "the church").
- Strategy: Plan-K, Execute-One 1. Plan: sample a K-step future action sequence from the diffusion model (denoising carves out plausible trajectories, not a single deterministic path). 2. Execute: run only the first action, observe new imagery. 3. Re-plan in a closed loop with fresh visual feedback.
- Stopping is treated as an explicit, independent decision, not a byproduct of action generation.
- Stop probability is estimated directly from the action logits at the diffusion model's all-masked initial state (i.e., before any action decisions), decoupling "should I stop?" from "where should I fly?"
- Result: less hovering near the goal, fewer premature stops and missed targets, better overall efficiency.
- Roughly +28% success rate in seen environments and +34% in unseen environments over prior best methods, with the lowest navigation error.
- The larger gains on unseen maps indicate strong generalization.
- Deng, Y., & Xu, F. (2026). *DreamFly: Causal Memory and Receding-Horizon Diffusion Planning for Aerial Vision-Language Navigation*. arXiv:2608.12308.
- Driess et al. (2023). *PaLM-E: An Embodied Multimodal Language Model*.
- Brohan et al. (2023). *RT-2: Vision-Language-Action Models*.
- Kim et al. (2025). *OpenVLA: An Open Source Vision-Language-Action Model*.
- Song et al. (2025). *Dream-VLA: Dreaming Visions for VLA*.
- Ho et al. (2020). *Denoising Diffusion Probabilistic Models*.
- Anderson et al. (2018). *Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments*.
- Chen et al. (2024). *OpenFly: A Benchmark for Aerial Vision-Language Navigation*.
Contribution 1: Causally aligned historical memory
Contribution 2: Receding-horizon diffusion planning (Plan-K, Execute-One)
Contribution 3: LiteStop — explicit, decoupled termination
Results on OpenFly
| Method | Test-Seen SR | Test-Unseen SR | Test-Seen SPL | Test-Unseen SPL | |---|---|---|---|---| | Prior best | ~25% | ~22% | ~20% | ~17% | | DreamFly | 32.04% | 29.46% | 28.22% | 23.54% |