Patch Policy: Robots Learn to See the World in Fragments

Forum topic · 小凯 · 2026-07-21

Summary

Patch Policy, from researchers at NYU and Meta AI (including Yann LeCun and Lerrel Pinto), challenges the common practice of compressing ViT features into a single global vector for robot control. Instead, it lets a lightweight Transformer policy directly consume dense, pre-trained ViT patch features (e.g., DINOv2, SAM), using a block-causal attention mask that allows full spatial attention within each timestep while preserving temporal causality. Evaluated across seven benchmarks (LIBERO, CALVIN, RLBench, MetaWorld, plus three real-world setups), Patch Policy achieves a 40% relative improvement over state-of-the-art policies using pooled global features and outperforms OpenVLA-OFT by 18% with only about 0.7% of its parameter count. The result suggests that visual feature quality can matter more than model scale in embodied control, enabling efficient, modular, real-time deployment on standard GPUs. Open questions include adaptive patch granularity, cross-modal fusion, long-horizon memory, and real-world robustness.

Key points

Problem: Most robot policies compress ViT image representations into a single global vector, discarding the fine spatial detail needed for precise manipulation. Using all hundreds of patches directly is too expensive for real-time control.
Method — Patch Policy: A lightweight Transformer policy directly consumes dense patch features from a frozen, pre-trained ViT (e.g., DINOv2 or SAM), combined with state inputs such as joint angles, and outputs actions.
Block-causal attention: Within a single timestep (one image), attention over patches is fully bidirectional; across timesteps, the mask enforces causality so the robot cannot 'see the future'. This preserves temporal causal structure while exploiting full spatial detail.
Results: Across four simulation benchmarks (LIBERO, CALVIN, RLBench, MetaWorld) and three real-world environments, Patch Policy achieves a 40% relative improvement over state-of-the-art policies using pooled global features.
Efficiency: Compared to the fine-tuned vision-language-action model OpenVLA-OFT, Patch Policy uses only about 0.7% of the parameters yet exceeds its performance by 18%.

Why dense patches beat global features

A global vector tells the robot *what* the scene roughly contains, but not *how* to act precisely — like teaching basketball by showing only a photo of a hoop. Dense patch features preserve edges, distances, and local geometry — the close-up details that manipulation requires.

Architecture

1. Frozen visual encoder: a pre-trained ViT extracts dense patch features offline-capable, with no task-specific retraining of the encoder. 2. Lightweight policy Transformer: takes patch features plus proprioceptive state and outputs action commands. 3. Block-causal attention mask: bidirectional attention within each frame, causal ordering across frames.

Broader implications

Democratization: small policy heads can run in real time on standard GPUs, avoiding billion-parameter VLA inference costs.
Modularity: as visual pre-training improves, the policy benefits for free by swapping encoders without retraining.
Efficiency over scale: the results suggest feature *quality* can matter more than parameter count in embodied control.

Open questions

Patch granularity: fixed patch sizes may not suit both fine grasping and coarse navigation; adaptive granularity is unexplored.
Cross-modal fusion: integrating touch, force, and audio with dense visual patches.
Long-horizon memory: standard Transformers may forget information from tens of seconds earlier; memory or state-space models (e.g., Mamba) may help.
Real-world robustness: lighting changes, occlusion, and dynamic objects can degrade patch features outside the lab.

Reference

Zhou, G., Cui, Z. J., Langford, A., Tan, B., LeCun, Y., & Pinto, L. (2026). Patch Policy: Efficient Embodied Control via Dense Visual Representations. arXiv:2607.18236.