Paper: When Validation Stops Learning: Auditing Update Admission for Continual Policy Learning

Forum topic · 小凯 · 2026-09-13

Summary

A new arXiv paper (2609.10873) by Qinzhen Ma and Ruihai Wu examines how independent validation in continual learning can reject harmful policy updates but also block useful learning. The authors argue that update admission should be evaluated through both error control and retained learning opportunities at a stated interaction budget. They identify a concrete failure mode: range-based confidence gates cannot certify that old-task behavior is unchanged even within substantial evaluation budgets. A standard paired-binomial construction reduces this burden when outcome disagreements between policies are rare. The paper also introduces certified historical-reference promotion and a round-level missed-opportunity metric. In a one-step pushing diagnostic with 32 seeds, fresh paired checks admitted 31.6% of a common update stream at 2,000 episodes per stage, while the range-based gate admitted none; unconditional replay still learned better in closed-loop runs. A learned-dynamics stress test further separates model bias from feedback-selection error. Physical-robot and VLA validation remain open problems.

Paper Overview

Research area: cs.AI Authors: Qinzhen Ma, Ruihai Wu Published: 2026-09-13 arXiv: 2609.10873

Abstract (translated)

Independent evaluation can reject harmful policy updates, yet it can also prevent useful continual learning. We argue that update admission must be assessed through both error control and retained learning opportunities at a stated interaction budget. We identify a concrete failure: a range-based confidence gate cannot certify unchanged old-task behavior within otherwise substantial budgets. A standard paired-binomial construction reduces this burden when outcome disagreements are rare. We also specify certified historical-reference promotion and a round-level missed-opportunity metric. In a constructed one-step pushing diagnostic with 32 seeds, fresh paired checks admit 31.6% of a common update stream at 2,000 episodes per stage, versus zero for the range-based gate; unconditional replay nevertheless learns better in closed-loop runs. The contribution is an admission-audit protocol with analytical and synthetic evidence; physical-robot and VLA validation remain open.

Original Abstract

Independent evaluation can reject harmful policy updates yet also prevent useful continual learning. We argue that update admission must be assessed through both error control and retained learning opportunities at a stated interaction budget. We identify a concrete failure: a range-based confidence gate cannot certify unchanged old-task behavior within otherwise substantial budgets. A standard paired-binomial construction reduces this burden when outcome disagreements are rare. We also specify certified historical-reference promotion and a round-level missed-opportunity metric. In a constructed one-step pushing diagnostic with 32 seeds, fresh paired checks admit 31.6% of a common update stream at 2,000 episodes per stage, versus zero for the range-based gate; unconditional replay nevertheless learns better in closed-loop runs. A separate learned-dynamics stress test distinguishes model bias from feedback-selection error. The contribution is an admission-audit protocol with analytical and synthetic evidence; physical-robot and VLA validation remain open.

Link: https://arxiv.org/abs/2609.10873