Paper: Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics

Forum topic · 小凯 · 2026-06-04

Summary

A new arXiv paper (2606.03982) investigates how language models (LMs) compare quantities that include measurement units, such as 110 cm versus 1.2 m, which requires combining a numeral with a symbolic unit scale. In controlled experiments spanning multiple unit systems, the authors find that accuracy degrades near the comparison boundary, where small changes in value flip the correct answer. The errors are systematic rather than random: linear surrogate models can predict LM preferences from numerical-difference and unit-scale-difference cues, and causal interventions on subspaces aligned with these variables shift model outputs. The authors conclude that LMs compare quantities through a bag of heuristics operating over numerals and units, rather than first converting both expressions into an exact shared-scale representation. The work offers insights into LMs' numeric reasoning limits and their internal representations of units and magnitudes. The paper is authored by Mutsumi Sasaki, Go Kamoda, Ryosuke Takahashi, Kosuke Sato, Kentaro Inui, Keisuke Sakaguchi, and Benjamin Heinzerling, and was released on June 2, 2026.

Paper Overview

Field: NLP Authors: Mutsumi Sasaki, Go Kamoda, Ryosuke Takahashi, Kosuke Sato, Kentaro Inui, Keisuke Sakaguchi, Benjamin Heinzerling Published: 2026-06-02 arXiv: 2606.03982

Abstract (Original)

Quantities with measurement units, such as 110 cm and 1.2 m, require language models (LMs) to combine a numeral with a symbolic unit scale. Here, we study how LMs compare such quantities in controlled settings spanning several unit systems. We find that accuracy degrades near the comparison boundary, where small changes in value determine the correct answer. The resulting errors are systematic: linear surrogate models predict LM preferences from numerical-difference and unit-scale-difference cues, and causal interventions on subspaces aligned with these variables shift model's output. The results suggest that LMs compare quantities through a bag of heuristics over numerals and units, rather than first converting both expressions to an exact shared-scale representation.

Key Takeaways

Task: Comparing quantities with measurement units (e.g., 110 cm vs. 1.2 m), which requires binding numerals to symbolic unit scales.
Setting: Controlled experiments across several unit systems.
Finding 1: Accuracy drops near the comparison boundary, where small value changes determine the correct answer.
Finding 2: Errors are systematic — linear surrogate models predict LM preferences using numerical-difference and unit-scale-difference cues.
Finding 3: Causal interventions on subspaces aligned with these variables shift model outputs, confirming the role of these cues.
Conclusion: LMs rely on a bag of heuristics over numerals and units rather than converting expressions to an exact shared-scale representation before comparison.

--- *Auto-collected on 2026-06-04.*