Planning to Learn: Why Exact Policy Gradient Loses to Cross-Entropy in Classification

Forum topic · 小凯 · 2026-10-06

Summary

A new arXiv paper (2610.03667) by Ian Osband explains why exact policy gradient underperforms cross-entropy in classification, despite classification being an ideal policy-gradient setting with no exploration, credit assignment, or action-sampling noise. The key insight is that the exact policy gradient is myopic: it evaluates each update only by its immediate gain in expected accuracy, ignoring that each update sets the starting point for future updates. Cross-entropy is reframed as "patient accuracy" — the total error an example would pay if its log-odds rose at unit rate forever. Truncating this total at the remaining amount of learning yields the horizon loss, a one-line modification that smoothly transitions from cross-entropy to exact policy gradient as training budget is exhausted. In a simple allocation model, horizon loss provably escapes local traps that capture both endpoints. Experiments on MNIST and ImageNet (ResNet-50, ResNet-101, ViT-S/16) show improved top-1 accuracy under flat learning rates, with gains growing as label noise increases.

论文概要

研究领域: ML 作者: Ian Osband 发布时间: 2026-10-02 arXiv: 2610.03667

English Summary

Policy-gradient methods are central to modern reinforcement learning, including LLM post-training. When they struggle, the usual suspects are exploration, credit assignment and action-sampling noise. Classification has none of them. A classifier is a policy whose expected reward, its *expected accuracy*, is the probability it assigns to the correct label, and because that label is known, the policy gradient is exact and smooth. Yet exact policy gradient loses to cross-entropy, even on expected accuracy.

The exact gradient is myopic: it values an update only by what it buys now, but each update also sets where the next one starts, so an update's value depends on how much learning remains. Viewed this way, cross-entropy is *patient accuracy* — the total error an example would pay if its log-odds rose at unit rate forever — while the exact policy gradient is the zero-horizon limit. Truncating this total at the remaining amount of learning gives the horizon loss: a one-line change that moves from cross-entropy toward the exact policy gradient as the budget for training is spent.

Key Results

In a simple allocation model, horizon loss provably escapes traps that capture both endpoints (exact policy gradient and cross-entropy).
On MNIST and ImageNet (ResNet-50, ResNet-101, and ViT-S/16), horizon loss improves top-1 accuracy under flat learning rates.
The gains grow as label noise increases.