Measuring Model Scale with Knowledge: Reverse-Engineering True LLM Parameter Counts from Black-Box APIs

An analysis of the paper *Incompressible Knowledge Probes* (Bojie Li, Pine AI, 2026): as reasoning benchmarks saturate collectively, factual knowledge becomes the only incompressible yardstick of scale.

📄 Paper: Bojie Li · Pine AI · 2026 | 📊 188 models · 27 vendors

1. Why a New Parameter Estimation Method?

Frontier labs no longer disclose real parameter counts. The mainstream alternative — "inference economics" (inferring scale from API throughput, pricing, and hardware costs) — carries inherent 2×+ uncertainty due to external variables like hardware generation, batching strategy, and quantization.

> 🌐 Key tension: Reasoning benchmarks (MMLU, GPQA, HELM) are collectively "saturating" — not because models stopped improving, but because procedural capabilities are compressible (a 2026 7B model can match a 2023 70B model at reasoning). Traditional benchmarks are losing their signal about model scale.

2. Core Idea: Incompressible Knowledge

The paper splits model parameters into three functional partitions:

| Partition | Meaning | Compressible? | |------|----------|----------| | N_fact | Factual knowledge (entity attributes, dates, names) | ❌ No | | N_proc | Procedural abilities (reasoning, instruction-following, tool use) | ✅ Yes | | N_ling | Linguistic ability (syntax, vocabulary, register) | ✅ Yes |

The "Densing Law" (capability density doubling roughly every 3.5 months) makes procedural skills increasingly efficient, but factual knowledge is different: a fact like "USTC Hackergame was founded in 2014" cannot be derived from general knowledge — it must be explicitly stored. Per Allen-Zhu & Li (2025), Transformers store roughly 2–4 bits of factual knowledge per parameter.

3. Experiment Design: A Seven-Tier Difficulty Ladder

The IKP benchmark contains 1,400 factual probes divided into 7 tiers by entity rarity on the web:

| Tier | Definition | Param range | Example | |------|------|----------|----------| | T1 | Universal knowledge | 0.1B–0.5B | Capital of Norway? | | T2 | Common references | 0.5B–7B | Who composed the Enigma Variations? | | T3 | Domain knowledge | 7B–32B | Year of the Battle of Hastings? | | T4 | Obscure knowledge | 32B–235B | Peter Druschel's research field? | | T5 | Deep knowledge | 235B–1T | Founding year of Eliot, Maine? | | T6 | Long-tail knowledge | 1T–10T | Jeffrey Helt's research field? | | T7 | Extreme tail | >10T | Which country is Tadekho Hill in? |

T1–T2 probes are mostly LLM-generated; T3–T7 sample real entities from Wikidata and CS academic databases (DBLP/OpenAlex), audited over 10 rounds to ensure no one can guess or reason their way to the answer.

4. Six Key Findings

1. R² = 0.917: Knowledge capacity is strictly log-linear with parameter count

Across 89 open models with known parameters (135M to 1600B), IKP accuracy is highly linear in log parameters. Every 10× increase in parameters yields about 14.7 percentage points of accuracy — a relationship spanning four orders of magnitude and 19 vendors.

2. Densing Law falsified for knowledge: facts don't compress over time

On 96 open models with release dates, IKP's time coefficient is -0.0010/month (95% CI: [-0.0031, +0.0008]) — statistically indistinguishable from zero. The Densing Law's predicted +0.0117/month growth is rejected at p<10⁻¹⁵. A fixed-size model from 2023 or 2026 knows the same number of facts.

3. The MoE secret: total parameters, not active ones

For mixture-of-experts models, total parameters predict knowledge capacity far better (R²=0.79) than active parameters (R²=0.51). Factual knowledge is distributed across all expert weights, not concentrated in the few activated per inference.

4. Knowledge fingerprints: lineage vs. independent training

If two models give the same wrong answer on the same rare T5–T6 fact (Hallucination Similarity, HSS), they likely share underlying weights. Three metrics cleanly separate model pairs:

• Shared weights: HSS ≥ 0.30, Jaccard ≥ 0.60
• Same family fine-tune: 0.10 ≤ HSS < 0.30, Jaccard ≥ 0.50
• Independently trained: HSS < 0.10 (even between adjacent version numbers)

5. The T7 cliff: all models drop to zero

On the hardest tier T7, only 2 of 188 models scored above 1%. Every frontier closed model — GPT-5 Pro, Claude Opus 4.7, Gemini 3.1 Pro — scored 0.0%. This is not an approaching ceiling but a structural cliff: the absolute long tail of knowledge grows faster than any current pretraining corpus.

6. The "silence tax" of safety alignment: knowing but not saying

Claude Sonnet 4 scores 6.7 points below Sonnet 3.7 — but its T5-tier refusal rate jumped from 54% to 88%. The model isn't missing the answer; safety policy forbids saying it. IKP scoring accounts for this: confidently wrong answers score -1, refusals score 0, rewarding honest conservatism.

5. Parameter Estimates for Closed Models (Excerpt)

| Model | Vendor | Accuracy | Est. Size | 90% CI | |-------|--------|----------|-----------|--------| | GPT-5.5 | OpenAI | 71.9% | ~9.7T | 3.2–28.7T | | Claude Opus 4.6 | Anthropic | 68.0% | ~5.3T | 1.8–15.6T | | GPT-5 Pro | OpenAI | 66.5% | ~4.1T | 1.4–12.2T | | Claude Opus 4.7 | Anthropic | 66.4% | ~4.0T | 1.4–12.0T | | o1 | OpenAI | 65.4% | ~3.5T | 1.2–10.3T | | Grok-4 | xAI | 64.8% | ~3.2T | 1.1–9.4T | | GPT-4o | OpenAI | 55.3% | ~720B | 241B–2.1T |

⚠️ Note: 90% prediction intervals are roughly ±3×. Heavily safety-aligned models like Claude Haiku "know but refuse," leading to underestimates (treat as lower bounds). The paper stresses these are "effective capacities," not exact physical parameter counts.

6. Marginal Gains from Chain-of-Thought

Across 27 base/thinking model pairs, thinking mode added an average of 2.2 points, peaking at T3–T4 but dropping to zero at T7. Strong evidence that chain-of-thought aids knowledge retrieval, not the creation of new knowledge.

7. Methodology and Limitations

• Sparse calibration: Above 1T, only DeepSeek V4 Pro (1.6T) and Kimi K2.5/K2.6 (~1T) serve as calibration points; high-end extrapolation rests on very few points.
• Safety-alignment contamination: Heavily RLHF'd models systematically underreport knowledge capacity.
• Probe contamination risk: If probes leaked into training data, estimates would be inflated — hence methodology is public but exact questions are withheld.
• Landmark-model circularity: The 6 "landmark models" defining tier boundaries have constructed scores at their tiers and are excluded as estimation targets.

💡 Commentary

The most notable contribution is not the headline number of "~9.7T for GPT-5.5," but establishing knowledge capacity as an independent, incompressible model dimension. When procedural benchmarks fail collectively due to the Densing Effect, IKP's incompressibility gives it a fundamental methodological advantage.

The AI community has long said "knowledge is parameters" — this paper turns that into a precisely measurable engineering reality expressed as an R²=0.917 log-linear scale. And HSS (identical errors) as a zero-shot model-lineage signal is as clever as any architectural watermarking scheme.

The ±3× confidence intervals remind us this is still "remote sensing" through a black box. But the direction is clear: in an era when everything can be compressed, the incompressible becomes the final yardstick for measuring everything.

---

📚 Paper: Bojie Li (Pine AI), *Incompressible Knowledge Probes*, 2026. 🔗 Code: github.com/19PINE-AI/ikp | 🌐 Interactive site: 01.me/research/ikp