The Core Idea
Large language models learn math the same way they learn language: by predicting the next token. The standard cross-entropy loss treats the token '7' following '3' exactly like any other token — the penalty for predicting '36' instead of '37' is identical to predicting an unrelated word. This ignores the fact that numbers are ordered: being off by 1 is meaningfully different from being wrong entirely. Standard MLE gives the model no inductive bias like "37 and 38 are close," so it must memorize exact token sequences for every number.
A team at The Hong Kong Polytechnic University (Zheng et al.) proposes DEL: Digit Entropy Loss for Numerical Learning of Large Language Models (arXiv:2605.20369), a loss function purpose-built for numerical learning.
Why Prior Fixes Fall Short
The paper analyzes two previous approaches within a unified criterion–distance framework:
- Number Token Loss (NTL): adds digit-distance penalties so predicting '35' instead of '37' costs less than '99'. Problem: the distribution becomes too sharp, harming language fluency since numbers appear embedded in natural text.
- Discretized Distance Loss (DDL): assigns probability mass to nearby digits. Problem: the distribution becomes too flat, giving unreasonable probability to distant digits.
- Overall digit prediction accuracy across all number positions, not just final answers
- Numerical distance of errors: wrong answers are systematically closer to the truth — critical in multi-step reasoning where small intermediate errors get amplified
- Architecture interaction: DEL changes the gradient signal, not the architecture; if the Transformer caps numeric representation, gains may plateau (scaling behavior unmeasured).
- Mixed text–number contexts: evaluation is limited to pure math benchmarks; real-world text interleaves numbers with words ("3 apples," "37.2 degrees").
- Digit-length limits: per-digit conditionals form long chains for very large numbers; scaling behavior is undiscussed.
- Training cost: per-digit binary cross-entropy is more expensive than standard cross-entropy; overhead at pretraining scale is unquantified.
- Synergy with reasoning methods: DEL is a pure supervised loss; its combination with chain-of-thought and RL-based approaches (e.g., GRPO) remains unexplored.
Both methods pull on the same rope — one over-tightens the distance term, the other loosens the criterion term — and neither finds the balance.
How DEL Works
DEL's insight is to drop the distance term entirely and instead optimize certainty at each digit position:
1. Per-digit conditional probabilities: a two-digit number like '37' is decomposed into conditional judgments on the tens and ones places, exploiting the positional structure of the Arabic numeral system. 2. Supervised entropy optimization via binary cross-entropy: naive entropy minimization would collapse to predicting a single digit; DEL uses digit labels to steer which direction entropy should converge. 3. Floating-point support: the decimal point is treated as a learnable "digit position," extending DEL from integers to the full floating-point domain — a unique contribution among these methods.
When the model gets each digit position conditionally right, distance information is encoded implicitly in the conditional probability structure — no explicit distance penalty needed.
Results
On seven mathematical reasoning benchmarks (arithmetic, algebra, word problems) across four LLMs (CodeLlama, Mistral, DeepSeek, Qwen-2.5), DEL consistently beats NTL and DDL on:
Floating-point support proved unexpectedly valuable: baselines often fail at simply "where to put the decimal point," while DEL's per-digit structure makes decimal placement a learnable decision.
Open Questions
References
1. Zheng, Z. et al. (2026). *DEL: Digit Entropy Loss for Numerical Learning of Large Language Models.* arXiv:2605.20369. 2. Rafailov, R. et al. (2023). *Direct Preference Optimization.* NeurIPS 2023. 3. Vaswani, A. et al. (2017). *Attention Is All You Need.* NeurIPS 2017. 4. Shao, Z. et al. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning.* arXiv:2402.03300. 5. Lewkowycz, A. et al. (2022). *Solving Quantitative Reasoning Problems with Language Models.* NeurIPS 2022. 6. Wang, X. et al. (2023). *Self-Consistency Improves Chain of Thought Reasoning in Language Models.* ICLR 2023.