Suppose you ask a large language model to rate a batch of social media posts for "toxicity" on a 1–5 scale. Your colleague uses the same model and the same posts, but with a −10 to +10 scale. Will your results match?
The answer: most likely not. And not by a small margin—about 20% of posts end up in different toxicity categories.
This is the problem revealed by the paper *Change the Number, Change the Answer: Assessing Rating Scale Sensitivity in LLM-Based Text Scoring*. A team from New York University and partner institutions systematically tested a long-overlooked question: how much do LLM scores change when you alter the design of the rating scale you give the model?
Experimental Design: Crossing Three Dimensions
The team designed a fairly comprehensive experiment, crossing three dimensions of variation:
- Scale format: continuous (e.g., 0.00 to 1.00) vs. ordinal (e.g., 1, 2, 3, 4, 5)
- Scale range: 3, 5, 11, or 21 points
- Scale sign: signed (e.g., −5 to +5) vs. unsigned (e.g., 0 to 10)
- Numerical sensitivity: changes in standardized scores. For example, a post scores 4 on a 5-point scale (standardized to 0.75) but 8 on an 11-point scale (standardized to 0.70). That 0.05 shift is numerical sensitivity.
- Semantic sensitivity: changes in score interpretation. "4 out of 5" might mean "fairly high," while "8 out of 11" might mean "moderate." Even when standardized values are close, their practical meaning can differ.
- Report scale sensitivity. When using LLM scoring, test at least 2–3 different scale designs and report the range of variation in results.
- Watch out for unipolar constructs. Unidimensional constructs like toxicity and credibility are more scale-sensitive and require extra caution.
- Don't over-trust rubrics. A detailed rubric is no substitute for a scale sensitivity check.
- Standardized reporting. In papers, report the format, range, and sign of your scale so readers can assess comparability.
- Accuracy hiding two independent dimensions: memorization vs. generalization
- Sentence-frequency metrics hiding appropriate vs. inappropriate uses
- Terminal rewards hiding the quality of each memory operation
- Soundness hiding both execution errors and task errors
These dimensions yield 2×4×2 = 16 scale designs. They tested three tasks: political leaning, sentiment, and toxicity scoring. Two prompting strategies (zero-shot vs. rubric-based) and five different LLMs were crossed in as well.
In total, they analyzed scoring results across 16 × 3 × 2 × 5 = 480 combinations.
Core Findings: Change One Design Element, 20% of Texts Change Category
The results are unsettling:
On average, changing any single element of scale design shifts standardized scores by about 0.10. That sounds small, but the practical impact is large—roughly 20% of texts get reclassified into different categories.
Imagine running a toxicity analysis on a 1–5 scale and finding a post is "moderately toxic." Re-run it on a 1–11 scale—same model, same posts—and it may now be classified as "low" or "high" toxicity. 20% of texts undergo such category jumps.
Toxicity is the most affected task. It is the only unipolar construct among the three tasks (toxicity only has degrees of "more or less," unlike political leaning, which spans left–right, or sentiment, which spans positive–negative). Unipolar constructs are more sensitive to scale design, possibly because without a "negative direction" anchor the model relies more on the scale itself for positioning.
Rubrics do not solve the problem. You might think that giving the model a detailed scoring rubric would stabilize its judgments. But the experiments show no significant difference in sensitivity between rubric-based and zero-shot prompting. The effect of scale design exceeds the effect of prompting strategy.
Two Kinds of Sensitivity: Numerical vs. Semantic
The paper's clearest contribution is distinguishing two kinds of sensitivity:
These two sensitivities are independent. You can have high numerical but low semantic sensitivity (numbers change but categories don't), or low numerical but high semantic sensitivity (numbers barely change but texts near category boundaries jump).
Why This Matters
This paper touches on a concept I have been tracking: scalar illusion.
What is scalar illusion? When we use a scalar (a single number) to measure a complex construct, we systematically ignore the uncertainty behind that number. We assume "toxicity score 0.73" is a precise measurement, when in reality it could be any value between 0.65 and 0.81—depending on the scale you use.
This is not unique to LLMs. Psychometrics has long known that scale design affects results. But LLMs make the problem more insidious, because:
1. LLMs look confident. They never say "I'm not sure whether this is 0.73 or 0.81"—they just give you a number, along with plausible-sounding explanations. 2. LLM scores are treated as objective measurements. In computational social science, researchers increasingly rely on LLM scores for text, then run statistical analyses on them. If the scores themselves are unstable, downstream statistical inference is built on sand. 3. Robustness checks are rare. Few studies check before publication whether "if we change the scale, do the conclusions still hold?"
Practical Recommendations
The paper offers several practical suggestions:
The Bigger Picture
This paper is another instance in the lineage of "scalar illusion." Over the past few months we have seen too many similar patterns:
Every time we have an LLM assign a scalar score to a complex construct, we perform an act of information compression—squeezing a high-dimensional, uncertain distribution into one number. The information lost in that compression is often more than what is retained.
Next time you see a result like "LLM score: 0.73," ask: what if we changed the scale?
---
Paper: https://arxiv.org/abs/2610.22187 Code repository: https://github.com/nyuolab/model_sensitivity