Three-Step Nav: A Hierarchical Global-Local Planner for Zero-Shot Vision-Language Navigation

Forum topic · 小凯 · 2026-05-01

Summary

Three-Step Nav (arXiv:2504.20756) is a zero-shot vision-language navigation (VLN) planner from Wanrong Zheng, Yunhao Ge, and Laurent Itti that improves MLLM-driven navigation agents without gradient updates or task-specific fine-tuning. Current zero-shot VLN agents built on multimodal large language models tend to drift off course, stop prematurely, and achieve low success rates. The method counters these failures with a three-view protocol: 'look forward' extracts global landmarks and sketches a coarse plan; 'look now' aligns the current visual observation with the next sub-goal for fine-grained guidance; and 'look backward' audits the entire trajectory to correct accumulated drift before stopping. The hierarchical global-local planner plugs into existing VLN pipelines with minimal overhead and achieves state-of-the-art zero-shot performance on the R2R-CE and RxR-CE datasets. Code is open-sourced at https://github.com/ZoeyZheng0/3-step-Nav.

Paper Overview

Field: Computer Vision Authors: Wanrong Zheng, Yunhao Ge, Laurent Itti Published: 2025-04-30 arXiv: 2504.20756

Abstract

Breakthrough progress in vision-based navigation through unknown environments has been achieved by using multimodal large language models (MLLMs). These models can plan a sequence of motions by evaluating the current view at each time step against the task and goal given to the agent. However, current zero-shot Vision-and-Language Navigation (VLN) agents powered by MLLMs still tend to drift off course, halt prematurely, and achieve low overall success rates.

We propose Three-Step Nav to counteract these failures with a three-view protocol:

1. "Look forward" — extract global landmarks and sketch a coarse plan. 2. "Look now" — align the current visual observation with the next sub-goal for fine-grained guidance. 3. "Look backward" — audit the entire trajectory to correct accumulated drift before stopping.

Key Points

Hierarchical global-local planner combining coarse global planning with fine-grained local guidance and trajectory auditing.
Requires no gradient updates or task-specific fine-tuning.
Plugs into existing VLN pipelines with minimal overhead.
Achieves state-of-the-art zero-shot performance on the R2R-CE and RxR-CE datasets.

Resources

Code: https://github.com/ZoeyZheng0/3-step-Nav
Paper: https://arxiv.org/abs/2504.20756