Scale Up Strategically: Diagnosing Instruction Factor Bias for Compositional Generalization in Robot Policies

Forum topic · 小凯 · 2026-07-26

Summary

This arXiv paper (2507.20476) by Yu Qi, Zhang Ye, and Xinyi Xu introduces a diagnostic framework for compositional generalization failures in language-conditioned robot policies. Pretrained policies often take shortcuts, over-relying on salient instruction cues instead of grounding language. The framework localizes these failures to individual instruction factors—reusable semantic components such as color, verb, object, size, and spatial attributes—and formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors. Two metrics quantify the bias: Factor Dominance Rate (FDR) for pairwise bias between factors and Factor Dominance Hierarchy (FDH) for a global ranking. Evaluation across six foundation policies reveals a broadly consistent ordering: color >= object >= spatial >= verb >= size, with color dominating and verbs and sizes least grounded. The diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget toward under-grounded factors outperforms baselines on simulation and real robots using half the demonstrations, enabling more efficient sampling and more generalizable policy learning.

Paper Overview

Research Area: Computer Vision / Robotics
Authors: Yu Qi, Zhang Ye, Xinyi Xu
Published: 2026-07-25
arXiv: 2507.20476

Abstract

Compositional generalization is essential for robots to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. The authors introduce a diagnostic framework that localizes this failure to individual instruction factors—reusable semantic components such as color, verb, object, size, and spatial attribute.

The framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics:

Factor Dominance Rate (FDR): captures pairwise bias between factors
Factor Dominance Hierarchy (FDH): aggregates pairwise biases into a global ranking

Evaluation on six foundation policies reveals a broadly consistent ordering:

> color >= object >= spatial >= verb >= size

Color dominates, while verbs and size are the least grounded factors.

Key Findings

The diagnostic is actionable: a bias-aware data collection strategy reallocates a fixed demonstration budget toward under-grounded factors.
This strategy outperforms baselines in both simulation and on real robots.
It achieves this with half the number of demonstrations, enabling more efficient sampling and more generalizable policy learning.

---

*Auto-collected on 2026-07-26.*