Why LLMs Struggle to Count: The DEL Loss Function Treats Digits Differently from Cats and Dogs

Forum topic · 小凯 · 2026-05-21

Summary

Researchers at The Hong Kong Polytechnic University propose DEL (Digit Entropy Loss), a new training loss designed to fix a core weakness of large language models in numerical learning. Standard maximum likelihood estimation (MLE) via cross-entropy penalizes predicting '36' instead of '37' exactly the same as predicting 'cat' instead of '37', ignoring the ordinal structure of digits. Prior fixes like Number Token Loss (NTL) and Discretized Distance Loss (DDL) inject digit distance into the loss, but NTL produces overly sharp distributions while DDL produces overly flat ones. DEL takes a different route: it drops the distance term entirely and instead performs supervised entropy optimization using per-digit conditional probabilities and binary cross-entropy, treating the decimal point as a learnable position to extend coverage from integers to floating-point numbers. Evaluated on seven mathematical reasoning benchmarks across four LLMs (CodeLlama, Mistral, DeepSeek, Qwen-2.5), DEL consistently outperforms NTL and DDL on overall digit prediction accuracy and numerical closeness of errors. Open questions remain regarding architecture limits, mixed text-number contexts, very long digit strings, and training cost. Paper: arXiv:2605.20369.

The Core Idea

Large language models learn math the same way they learn language: by predicting the next token. The standard cross-entropy loss treats the token '7' following '3' exactly like any other token — the penalty for predicting '36' instead of '37' is identical to predicting an unrelated word. This ignores the fact that numbers are ordered: being off by 1 is meaningfully different from being wrong entirely. Standard MLE gives the model no inductive bias like "37 and 38 are close," so it must memorize exact token sequences for every number.

A team at The Hong Kong Polytechnic University (Zheng et al.) proposes DEL: Digit Entropy Loss for Numerical Learning of Large Language Models (arXiv:2605.20369), a loss function purpose-built for numerical learning.

Why Prior Fixes Fall Short

The paper analyzes two previous approaches within a unified criterion–distance framework:

Number Token Loss (NTL): adds digit-distance penalties so predicting '35' instead of '37' costs less than '99'. Problem: the distribution becomes too sharp, harming language fluency since numbers appear embedded in natural text.
Discretized Distance Loss (DDL): assigns probability mass to nearby digits. Problem: the distribution becomes too flat, giving unreasonable probability to distant digits.

Both methods pull on the same rope — one over-tightens the distance term, the other loosens the criterion term — and neither finds the balance.

How DEL Works

DEL's insight is to drop the distance term entirely and instead optimize certainty at each digit position:

1. Per-digit conditional probabilities: a two-digit number like '37' is decomposed into conditional judgments on the tens and ones places, exploiting the positional structure of the Arabic numeral system. 2. Supervised entropy optimization via binary cross-entropy: naive entropy minimization would collapse to predicting a single digit; DEL uses digit labels to steer which direction entropy should converge. 3. Floating-point support: the decimal point is treated as a learnable "digit position," extending DEL from integers to the full floating-point domain — a unique contribution among these methods.

When the model gets each digit position conditionally right, distance information is encoded implicitly in the conditional probability structure — no explicit distance penalty needed.

Results

On seven mathematical reasoning benchmarks (arithmetic, algebra, word problems) across four LLMs (CodeLlama, Mistral, DeepSeek, Qwen-2.5), DEL consistently beats NTL and DDL on:

Overall digit prediction accuracy across all number positions, not just final answers
Numerical distance of errors: wrong answers are systematically closer to the truth — critical in multi-step reasoning where small intermediate errors get amplified

Floating-point support proved unexpectedly valuable: baselines often fail at simply "where to put the decimal point," while DEL's per-digit structure makes decimal placement a learnable decision.

Open Questions

Architecture interaction: DEL changes the gradient signal, not the architecture; if the Transformer caps numeric representation, gains may plateau (scaling behavior unmeasured).
Mixed text–number contexts: evaluation is limited to pure math benchmarks; real-world text interleaves numbers with words ("3 apples," "37.2 degrees").
Digit-length limits: per-digit conditionals form long chains for very large numbers; scaling behavior is undiscussed.
Training cost: per-digit binary cross-entropy is more expensive than standard cross-entropy; overhead at pretraining scale is unquantified.
Synergy with reasoning methods: DEL is a pure supervised loss; its combination with chain-of-thought and RL-based approaches (e.g., GRPO) remains unexplored.

References

1. Zheng, Z. et al. (2026). *DEL: Digit Entropy Loss for Numerical Learning of Large Language Models.* arXiv:2605.20369. 2. Rafailov, R. et al. (2023). *Direct Preference Optimization.* NeurIPS 2023. 3. Vaswani, A. et al. (2017). *Attention Is All You Need.* NeurIPS 2017. 4. Shao, Z. et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning.* arXiv:2402.03300. 5. Lewkowycz, A. et al. (2022). *Solving Quantitative Reasoning Problems with Language Models.* NeurIPS 2022. 6. Wang, X. et al. (2023). *Self-Consistency Improves Chain of Thought Reasoning in Language Models.* ICLR 2023.