Bias-Aware Evaluation and Data Collection for Compositional Generalization in Robotic Manipulation (arXiv 2507.21742)

Forum topic · 小凯 · 2026-07-27

Summary

This paper introduces a diagnostic framework for compositional generalization in robotic manipulation, addressing the tendency of pretrained policies to take shortcuts by relying on salient instruction cues instead of grounding language. The framework localizes failures to individual instruction factors—color, verb, object, size, and spatial attributes—and formalizes instruction factor bias with two metrics: Factor Dominance Rate (FDR), which captures pairwise bias between factors, and Factor Dominance Hierarchy (FDH), which aggregates these into a global ranking. Evaluation across six foundation policies reveals a broadly consistent ordering of color >= object >= spatial >= verb >= size, with color dominating and verbs and size least grounded. Building on this diagnosis, the authors propose a bias-aware data collection strategy that reallocates a fixed demonstration budget toward under-grounded factors. On both simulated and real robots, this approach outperforms baselines while using only half the demonstrations, enabling more sample-efficient and generalizable policy learning.

Paper Overview

Field: Computer Vision Authors: Yu Qi, Zhang Ye, Xinyi Xu Published: 2025-07-27 arXiv: 2507.21742

Key Contributions

Problem: Compositional generalization is essential for robots to follow diverse instructions, but pretrained policies are known to take shortcuts—deferring to salient cues rather than grounding language.
Diagnostic framework: Localizes this failure to individual instruction factors, i.e., reusable semantic components such as color, verb, object, size, and spatial attributes.
Formalization: Introduces *instruction factor bias*, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, quantified via two metrics:
Factor Dominance Rate (FDR): captures pairwise bias between factors.
Factor Dominance Hierarchy (FDH): aggregates pairwise biases into a global ranking.
Findings: Evaluation on six foundation policies reveals a broadly consistent ordering: color >= object >= spatial >= verb >= size, with color dominant and verbs and size the least grounded factors.
Actionable diagnosis: A bias-aware data collection strategy reallocates a fixed demonstration budget toward under-grounded factors.
Results: In simulation and on real robots, the approach outperforms baselines while using only half of the demonstrations, enabling more sample-efficient and generalizable policy learning.

Original Abstract (excerpt)

Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual instruction factors, e.g., reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, i.e., color ... (excerpt)

Full paper: arxiv.org/abs/2507.21742