DelTA: Discriminative Token Credit Assignment Fixes RLVR's Reward Problem

Forum topic · 小凯 · 2026-05-24

Summary

Reinforcement Learning from Verifiable Rewards (RLVR) trains math-reasoning models by rewarding correct final answers, but standard policy-gradient updates credit every token equally. Because answers are full of high-frequency formatting tokens like line breaks and brackets, these common features dominate the gradient update while the sparse, genuinely useful reasoning tokens get diluted. A new paper, DelTA (Discriminative Token Credit Assignment, arXiv:2605.21467), reframes policy-gradient updates as a linear classifier: it amplifies tokens discriminative of good answers and suppresses tokens common to both good and bad ones, giving each token a specificity-based weight. Experiments on Qwen3-8B and Qwen3-14B across seven core math benchmarks show average gains of 3.26 points (8B) and 2.62 points (14B) over the strongest baselines, with consistent robustness on code generation and alternative model backbones. DelTA offers a cleaner solution to token-level credit assignment in RLVR.

DelTA: How AI Picks Breakthrough Steps Out of Noise in RLVR

The Problem: Rewarding the Whole Answer, Not the Insight

Training models to do math via "Reinforcement Learning from Verifiable Rewards" (RLVR) is like teaching a child archery: hit the bullseye, get a candy. But when a model produces a thousand-word solution, which step was the stroke of genius and which was filler?

Current algorithms often get this wrong. They see solutions full of line breaks and brackets and mistakenly credit these formatting tokens as heroes, while the truly decisive mathematical symbols get neglected. The result: undifferentiated rewards and slow learning.

Root Cause: Formatting Tokens Hijack the Gradient

RLVR's update rule is essentially:

\nabla_\theta J(\theta) \approx \sum_t A \cdot \nabla_\theta \log \pi_\theta(a_t | s_t)

Here $$A$$ is the advantage (score) and $\nabla_\theta \log \pi_\theta$ is each token's sensitivity direction. High score means every token on the path gets praised.

The flaw: both good and bad answers contain large amounts of boilerplate formatting tokens. When averaging, these high-frequency features become dominant, diluting the genuinely useful "sparse features" — as if the janitor got top honors while the general who won the battle went unrecognized.

The Fix: DelTA's Discriminative Credit Assignment

A new paper, DelTA (Discriminative Token Credit Assignment), breaks the deadlock with a simple idea: stop the blanket rewards, add a discriminator.

The researchers observed that a model's gradient updates implicitly contain a "linear classifier." Since it's classification, features unique to good answers should be amplified, while features common to all answers should be suppressed.

DelTA computes a weight coefficient for each token: high specificity gets a large coefficient; generic filler gets its coefficient crushed.

Results: Qwen3 Benchmarks

Tested on Qwen3-8B and Qwen3-14B base models across seven core math benchmarks:

8B scale: average score beats the strongest baseline by 3.26 points
14B scale: exceeds the strongest baseline by 2.62 points
DelTA remains robust on code generation tasks and with alternative model backbones

Clear rewards and punishments make for a well-trained model — DelTA brings transparency to the "credit assignment" accounting in LLM reinforcement learning.

References

Paper title: DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards
Release date: May 22, 2026
arXiv ID: arXiv:2605.21467
Core problem: Token-level credit assignment in RLVR
Mechanism: Reframing policy-gradient updates as a linear discrimination view to amplify discriminative features