What Matters in Quality-Aware VLA Pretraining: AXIS-Bench and Empirical Findings

Forum topic · 小凯 · 2026-10-15

Summary

This paper investigates how demonstration quality should be used during vision-language-action (VLA) pretraining to improve post-fine-tuning performance. The authors compare weighting, filtering, and conditioning strategies across reward definitions and label granularities, using 5,000 naturally suboptimal human demonstrations and 24 target tasks, with matched demonstrations and training budgets within each VLA. Across three VLAs, key findings include: approach-aware episode conditioning outperforms plain behavioral cloning; conditioning and filtering favor episode-level and chunk-level labels respectively, with model-dependent reversals; and constant-tag fine-tuning after conditioning-based pretraining improves mean success rates, regardless of whether tags are present at inference. The team introduces AXIS-Bench, built from crowdsourced simulation demonstrations, providing recorded object and contact states so quality labels can be recomputed on shared trajectories. Replayable trajectories support workspace-aligned rendering and user-defined cross-domain target tasks. Real-world experiments show simulation pretraining raises success rates from 9.6% to 31.7%.

Paper Overview

Field: Machine Learning arXiv: 2610.18869

Abstract

How should demonstration quality be used during vision-language-action (VLA) pretraining to improve performance after target-task fine-tuning? The authors compare weighting, filtering, and conditioning across reward definitions and label granularities using 5,000 naturally suboptimal human demonstrations and 24 target tasks. Within each VLA, demonstrations and training budgets are matched. Additional comparisons examine pretraining data size and the roles of quality tags during fine-tuning and inference.

Key Findings

Across three VLAs:

Approach-aware episode conditioning improves mean success over plain behavioral cloning.
Conditioning and filtering favor episode-level and chunk-level labels, respectively, on average — with model-dependent reversals.
Constant-tag fine-tuning after conditioning-based pretraining improves mean success, whether or not tags are available at inference time.

AXIS-Bench

To enable these controlled comparisons, the authors release AXIS-Bench, built from crowdsourced simulation demonstrations. It provides:

Recorded object and contact states, allowing quality labels to be recomputed on shared trajectories.
Replayable trajectories supporting workspace-aligned rendering.
User-defined cross-domain target tasks.

Real-World Results

In real-world experiments, simulation-based pretraining improved success rates from 9.6% to 31.7%.

---

*Auto-collected on 2026-10-15.*