Why LLMs Struggle with Counting: DEL Explains—and Fixes It

> Paper: DEL: Digit Entropy Loss for Numerical Learning of Large Language Models > Authors: Zhaohui Zheng, Chenhang He, Shihao Wang, Yuxuan Li, Ming-Ming Cheng, Lei Zhang > Institution: Hong Kong Polytechnic University (PolyU) > arXiv: 2605.20369 · Date: 2026-05-19 · Categories: cs.CL / cs.AI / cs.LG > Core claim: LLMs' weakness in digit prediction stems from MLE, which was never designed for numerical learning. Existing fixes make the output distribution either too sharp or too flat. DEL redesigns the objective as *supervised entropy optimization* using per-digit conditional probabilities and binary cross-entropy, drops the numeric distance term, and extends from integers to floating-point numbers.

Imagine teaching a five-year-old to read the number "37." They say "3," hesitate, then guess "7?"—unsure whether 7 ends the number or what should follow the 3. That uncertainty is exactly what every large language model experiences.

LLMs learn math the same way they learn language: next-token prediction. When a model sees "3," standard cross-entropy loss trains it to predict "7"—treating the distinction between "3" and "7" exactly like the distinction between "cat" and "dog."

But digits differ fundamentally from words: numbers are ordered. 37 is not "a 3 followed by a 7"—it is three tens plus seven ones. Predicting "36" instead of "37" is off by 1; there is no "off by 1" between "cat" and "dog." Standard training ignores this continuous structure entirely, forcing the model to memorize every number's exact token sequence with no inductive bias that "37 is close to 38."

Why MLE Makes You Bad at Counting

With MLE, if the model predicts "36" instead of "37," the penalty is identical to predicting "cat" instead of "37"—cross-entropy only asks *did you guess right?*, never *how far off were you?* This makes number learning extremely sample-inefficient.

Prior Fixes: Too Sharp or Too Flat

• Number Token Loss (NTL): adds numeric distance penalties—mistakenly predicting "35" for "37" costs less than predicting "99." Problem: the distribution becomes *too sharp*, making the model overconfident on digits, which hurts fluency since numbers are embedded in natural language contexts.
• Discretized Distance Loss (DDL): also uses distance, but spreads probability mass to neighboring numbers. Problem: the distribution becomes *too flat*, assigning implausible probability to distant numbers.

The paper unifies both in a criterion–distance formulation: the criterion term controls whether probability concentrates or disperses; the distance term injects geometric priors. NL overweights the distance term (too sharp); DDL is too loose on the criterion term (too flat). Neither finds the balance.

DEL: Drop Distance, Use Entropy

DEL's key insight: don't penalize how far off you were—optimize certainty at each digit position.

Three design choices:

1. Per-digit conditional probabilities. A two-digit number like "37" is learned as simultaneous judgments on the tens and ones positions, exploiting the positional structure of the decimal system. 2. Binary cross-entropy-guided entropy optimization. Unsupervised entropy minimization would collapse the model to always predict "0." DEL supervises entropy optimization with binary cross-entropy (whether each candidate digit matches the ground truth) to constrain its direction. 3. Extension to floating-point numbers. The decimal point is treated as an additional "digit position," so the model learns *when* to emit it—extending coverage from integers to the full floating-point domain.

Crucially, DEL abandons the distance term. In NTL and DDL, distance is the source of trouble. DEL's insight: you don't need to explicitly tell the model "37 and 38 are close"—when each digit is predicted correctly through conditional probabilities, distance information is naturally encoded in that structure.

Results

Across seven math reasoning benchmarks (arithmetic, algebra, word problems) with four LLMs (CodeLlama, Mistral, DeepSeek, Qwen-2.5), DEL consistently outperforms NTL and DDL:

• Overall prediction accuracy improves at *all* numeric positions, not just final answers.
• Numerical distance of errors shrinks: even wrong answers are systematically closer to the truth—critical in multi-step reasoning, where small intermediate errors get amplified.
• Floating-point support paid unexpected dividends: baselines often fail at the simple question of *where to put the decimal point*, while DEL's per-digit structure makes decimal placement a learnable decision.

Honest Limitations

• Loss–architecture interaction: DEL changes gradient signals, not architecture. If the Transformer's representational capacity for numbers is capped, DEL may hit a ceiling; scaling behavior isn't measured.
• Mixed text–number contexts: evaluation is on pure math benchmarks. Whether per-digit structure integrates seamlessly into general language tasks ("37.2 degrees," "221 BC") is untested.
• Digit-length limits: very long numbers create long per-digit conditional chains; scaling to huge numbers is undiscussed.
• Training efficiency: per-digit binary cross-entropy is costlier than standard cross-entropy; compute overhead is not analyzed in the abstract.
• Synergy with reasoning methods: whether DEL composes with chain-of-thought and RL-based approaches (e.g., GRPO on math rewards) remains unexplored.

References

1. Zheng, Z. et al. (2026). *DEL: Digit Entropy Loss for Numerical Learning of Large Language Models.* arXiv:2605.20369. 2. Rafailov, R. et al. (2023). *Direct Preference Optimization.* NeurIPS 2023. 3. Vaswani, A. et al. (2017). *Attention Is All You Need.* NeurIPS 2017. 4. Shao, Z. et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning.* arXiv:2402.03300. 5. Lewkowycz, A. et al. (2022). *Solving Quantitative Reasoning Problems with Language Models.* NeurIPS 2022. 6. Wang, X. et al. (2023). *Self-Consistency Improves Chain of Thought Reasoning in Language Models.* ICLR 2023.