Learning Action Priors for Cross-embodiment Robot Manipulation (arXiv 2606.19233)

Forum topic · 小凯 · 2026-06-26

Summary

A new robotics paper (arXiv 2606.19233, Dong Jing, Tianqi Zhang, Jiaqi Liu) proposes pretraining the action module of Vision-Language-Action (VLA) models with motion priors before cross-modal alignment. The two-stage framework first trains a lightweight flow-matching encoder-decoder action module on unlabeled action trajectories to learn temporal motion structure, without processing vision or language tokens. This learned prior is then transferred into VLA training via decoder reuse and early latent distillation, aligning vision-language features with the action embedding space while keeping end-to-end optimization. The trained encoder also compresses state-action history into a single temporal context token for cheap history-aware modeling. Experiments on 13 diverse cross-embodiment tasks in simulation and on real robots show faster convergence and higher success rates than VLA training without action priors, with notable gains on data-scarce real-world tasks. Scaling stage-one action data further improves downstream VLA performance.

Paper Overview

Field: Robotics Authors: Dong Jing, Tianqi Zhang, Jiaqi Liu Published: 2026-06-25 arXiv: 2606.19233

Abstract

Most Vision-Language-Action (VLA) models build on a Vision-Language Model (VLM) backbone by attaching an action module and optimizing the full policy jointly. This design inherits strong visual and linguistic priors from the VLM, but leaves the action module to learn physical motion almost from scratch. As a result, the policy lacks an explicit motion prior, forcing early optimization to simultaneously discover temporal action dynamics and cross-modal alignment, a challenge further amplified in cross-embodiment settings.

The authors propose pretraining the action module with motion priors before cross-modal VLA alignment, via a two-stage training framework that equips the action module with cross-embodiment temporal motion structure before VLA training begins:

Stage 1: A lightweight flow-matching-based encoder-decoder action module efficiently learns temporal motion structure from unconditional action trajectories alone, without processing vision or language tokens.
Stage 2: The learned prior is transferred into VLA training through decoder reuse and early latent distillation, aligning vision-language features with the action embedding space while still allowing end-to-end policy optimization.

Additionally, the trained encoder serves as a compact history compressor, summarizing state-action history into a single temporal context token for low-cost history-aware modeling.

Results

Extensive experiments across 13 diverse cross-embodiment tasks in simulation and on real platforms validate the approach:

Faster convergence and higher success rates compared to VLA training without action priors.
Significantly stronger performance on data-scarce real-world tasks.
Scaling stage-one action data yields more generalizable action priors that directly improve downstream VLA performance.

--- *Auto-collected on 2026-06-26*