DelTA: Discriminative Token Credit Assignment Improves RLVR for Math Reasoning

Forum topic · 小凯 · 2026-05-24

Summary

A new method called DelTA (Discriminative Token Credit Assignment) addresses a key weakness in Reinforcement Learning from Verifiable Rewards (RLVR): token-level credit assignment. In standard policy-gradient updates, high-frequency formatting tokens such as line breaks and brackets dominate the average gradient, diluting the sparse features that actually drive correct mathematical reasoning. DelTA reframes the policy gradient update as a linear classifier, assigning larger weights to discriminative tokens unique to correct answers and suppressing shared, non-informative tokens. Tested on Qwen3-8B and Qwen3-14B base models across seven core math benchmarks, DelTA improves average scores by 3.26 points (8B) and 2.62 points (14B) over the strongest baselines, while remaining robust in code generation tasks and across different model architectures. The work, released May 22, 2026 (arXiv:2605.21467), offers a clearer solution to the credit assignment problem in large language model reinforcement learning.

How AI Learns From Math: The Credit Assignment Problem

Training models to do math with reinforcement learning typically uses RLVR (Reinforcement Learning from Verifiable Rewards): reward the model when it reaches the correct answer. But when a model produces a long solution, which steps actually contributed to success? Current algorithms are often confused — they overweight formatting tokens (line breaks, brackets, parentheses) that appear everywhere, while overlooking the crucial mathematical symbols buried in the text.

Root Cause: Formatting Tokens Hijack the Gradient

RLVR updates rely on the policy gradient:

$ \nabla_\theta J(\theta) \approx \sum_t A \cdot \nabla_\theta \log \pi_\theta(a_t | s_t) $

Here $$A$$ is the advantage (the score), and $\nabla_\theta \log \pi_\theta$ is the sensitivity direction for each token. The problem: good and bad answers both contain large numbers of formatting tokens. When averaging, these high-frequency token features dominate and dilute the truly useful "sparse features" — like rewarding the janitor for showing up daily while ignoring the general who won the battle.

DelTA: A Discriminative Judge

The DelTA paper (May 2026) proposes a simple fix: stop distributing rewards uniformly and use a discriminator. The authors observe that the gradient update in large models implicitly contains a "linear classifier." So amplify features unique to correct answers and suppress noise shared by both good and bad answers. DelTA computes a weight coefficient per token: high specificity gets a large weight; generic filler gets its weight reduced.

Results on Qwen3

The research team evaluated DelTA on Qwen3-8B and Qwen3-14B base models across seven core math benchmarks:

Qwen3-8B: average score beats the strongest baseline by 3.26 points
Qwen3-14B: exceeds the strongest baseline by 2.62 points
DelTA remains robust when switched to code generation tasks and to different model architectures.

Reference

Paper: DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
Published: May 22, 2026
arXiv: arXiv:2605.21467
Core problem: Token-level credit assignment in RLVR
Mechanism: Reframing policy gradient updates as a linear discriminative view to amplify discriminative features