Scale Up Strategically: Bias-Aware Evaluation and Data Collection for Compositional Generalization in Robotic Manipulation

Forum topic · 小凯 · 2026-07-27

Summary

This paper (arXiv 2507.21742) addresses compositional generalization in robotic manipulation, where pretrained policies take shortcuts by relying on salient cues instead of grounding language. The authors introduce a diagnostic framework that localizes these failures to individual instruction factors—reusable semantic components such as color, verb, object, size, and spatial attributes. They formalize instruction factor bias and quantify it with two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), which aggregates these into a global ranking. Evaluations across six foundation policies reveal a broadly consistent bias ordering of color >= object >= spatial >= verb >= size, with color dominating while verbs and size are least grounded. The diagnosis is also actionable: a bias-aware data collection strategy that reallocates a fixed demonstration budget toward under-grounded factors outperforms baselines on both simulation and real robots using only half the demonstrations, enabling more sample-efficient and generalizable policy learning.

Paper Overview

Field: Computer Vision (CV) Authors: Yu Qi, Zhang Ye, Xinyi Xu Published: 2025-07-27 arXiv: 2507.21742

Original Abstract

Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual instruction factors, e.g., reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, i.e., color >= object >= spatial >= verb >= size, where color dominates and verbs and size are the least grounded. We further show this diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget to under-grounded factors outperforms baselines on simulation and real robots with only half the demonstrations, enabling more sample-efficient and generalizable policy learning.

Key Contributions

Diagnostic framework that localizes compositional generalization failures to individual instruction factors (color, verb, object, size, spatial attributes).
Two metrics:
Factor Dominance Rate (FDR): captures pairwise bias between instruction factors.
Factor Dominance Hierarchy (FDH): aggregates pairwise biases into a global ranking.
Consistent bias ordering across six foundation policies: color >= object >= spatial >= verb >= size — color is the dominant shortcut; verbs and size are the least grounded.
Actionable data collection: reallocating a fixed demonstration budget toward under-grounded factors outperforms baselines with only half the demonstrations, in both simulation and on real robots.

Significance

The work shows that diagnosing language-grounding biases in fine-tuned policies enables strategically targeted data collection, leading to more sample-efficient and generalizable robotic manipulation policies.