WordSaladChopper: Ending Decode Waste in Reasoning Models
*Based on the EMNLP 2025 oral paper WordSaladChopper. Open-source code: GitHub.*
The problem: "word salad"
Large reasoning models (LRMs) excel at complex reasoning tasks, but at the cost of far more decoded tokens. A significant fraction of this expensive output is meaningless, repetitive, looping text—the "word salad" phenomenon—where the model restates earlier points, phrases, or paragraphs without advancing the reasoning.
Consequences:
- Resource waste: over 50% of the decoding budget can be consumed by word-salad tokens
- Higher cost: e.g., OpenAI o3 output tokens cost ~4x input tokens
- Worse UX: longer waits and degraded perceived quality
Key insight
When an LRM falls into a word-salad loop, its internal hidden states exhibit identifiable patterns—the model in some sense "knows" it is stuck in unproductive repetition. This makes real-time detection possible.
The WSC system
WSC is a plug-and-play, lightweight module that requires no modification or retraining of the underlying model:
1. Detect: a single-layer linear classifier continuously analyzes hidden states (primarily after <\n\n> tokens) to identify word-salad patterns.
2. Chop: once detected, the redundant generation sequence is immediately truncated.
3. Regenerate: carefully designed prompts steer the model back onto a productive reasoning track.
Training: positive and negative examples (fluent reasoning vs. repetitive loops) are collected automatically by having the target LRM reason on tasks; the classifier learns from the hidden-state vectors.
Strengths: low intrusion, ~0.4% runtime overhead, 10–57% length reduction with accuracy essentially preserved, real-time intervention.
Limitations: the detector must be trained per model (hidden-state distributions differ across models), effectiveness varies by model/task pair, detection relies heavily on key tokens like <\n\n>, and threshold tuning trades detection precision against intervention timing.
Experimental results (paper Table 8)
| Model | Dataset | Original Acc. | WSC Acc. | Length Change | |---|---|---|---|---| | Qwen-1.5B | GSM8K | 82.56% | 83.02% | -19.20% | | Qwen-1.5B | GPQA-Diamond | 35.86% | 35.35% | -26.73% | | Llama-8B | MATH-500 | 83.60% | 83.80% | -6.60% | | Llama-8B | AIME25 | 28.75% | 29.16% | -4.11% |
Accuracy is preserved (occasionally slightly improved) while generating substantially fewer tokens.
Implications
Given WSC's negligible computational cost and significant savings—combined with the low semantic value of word-salad tokens—the authors argue that WSC or similar components are a must-have for any user-experience-centric LRM application.
Key metrics: length reduction 10–57% · runtime overhead 0.4% · detection accuracy 92%+