The Quantization Trap: The Hidden Costs of 4-Bit Quantization
This infographic-style post from zhichai.net argues that the widespread assumption "lower precision = less VRAM and more efficiency" breaks down in real LLM deployments—especially for multi-hop reasoning.
The Myth vs. The Finding
- Common myth: Lower precision always saves memory and improves efficiency.
- Paper's finding: In multi-hop reasoning workloads, 4-bit models can actually consume more power and run slower.
- Trust (TSI): Reasoning accuracy — 4-bit leads to logic degradation.
- Economic (ESI): Throughput per cost — dequantization slows serving.
- System/Energy (SSI): Energy efficiency — wasted operations increase power draw.
Source: Han, H., Liu, X., et al. (2026). "The Quantization Trap: Breaking Linear Scaling Laws in Multi-Hop Reasoning."
Conversion Overhead Ratio (COR): Compute Eaten by "Unpacking"
On A100/H100 hardware, which does not support native 4-bit operations, a large share of compute is wasted on dequantization overhead:
| Workload Component | Share of Compute | |---|---| | Actual computation (FP16) | ~30% | | Conversion overhead (dequantization) | ~70% |
When COR > 1.0, the bottleneck is "unpacking the data," not inference itself.
Sustainability Index (SI) Framework
The post proposes evaluating quantization across three pillars:
Multi-Hop Reasoning and Logic Collapse
In agent workflows, small quantization errors compound like a snowball:
1. Tiny error introduced by quantization 2. False premise in an intermediate reasoning step 3. Logic collapse in the final output
Deployment Guide
| Zone | Scenarios | Notes | |---|---|---| | ✅ Safe zone | Single-turn chat, text summarization, simple retrieval | Low VRAM footprint, fast responses | | 🚫 Danger zone | Complex agent tasks, code generation, math reasoning — use 8/16-bit | Risks: confused logic, task failure, wasted compute |
> Don't let "saving VRAM" become the culprit that destroys your model's intelligence.