Infinite-Parameter LLMs: Compiling Knowledge into Weights Instead of Prompts

Forum topic · ✨步子哥 · 2026-09-17

Summary

A zhichai.net forum post analyzes the arXiv paper 2609.18842, 'Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data.' Conventional LLMs keep weights frozen after training, so all runtime data must live in the context window, causing linear cost growth per token and total knowledge loss after each session. The paper proposes a different architecture: replace the fixed Mixture-of-Experts expert bank with a frozen generator that maps a latent code z into a low-rank weight delta, plus a Bayesian belief distribution P(z) that updates per token. This compiles runtime evidence into weights at inference time without gradient descent, avoiding catastrophic forgetting. Benchmark results on five QA datasets show in-context learning wins on short, clean evidence (SQuAD: 85.3 vs 51.8), while weight compilation wins on long, noisy, multi-hop evidence (MS MARCO: 48.0 vs 33.6; MuSiQue: 45.3 vs 40.9). Belief accumulation routes more accurately and at constant per-token cost O(K) versus linear context growth. Limitations: the small generator cannot match a large MoE's knowledge capacity, benefits appear only with long evidence, and only discrete latent codes are implemented. The post discusses implications for MoE research, RAG, agent memory, and inference infrastructure.

This zhichai.net post examines a September 2026 arXiv paper (2609.18842) that proposes compiling runtime data into model weights instead of stuffing it into prompts.

The problem

Today's LLMs (GPT-5, Claude Opus 5, Gemini 3) freeze their weights after training. Every user message causes the model to re-encode the entire conversation history through the transformer layers. Two costs follow:

Computation grows linearly with context length — every generated token re-attends to everything.
Knowledge vanishes when the session ends — domain knowledge, preferences, and corrections live only in the prompt.

Prior attempts at inference-time weight updates (test-time training, fast weights) either produce brittle point estimates, suffer catastrophic forgetting, or are computationally impractical.

Key points

Generate experts, don't store them. Standard MoE (e.g., DeepSeek-V3's 671B/37B activated) uses a fixed expert bank. The paper replaces FFN layers with three components: a frozen base FFN, a frozen generator mapping a latent code z to a low-rank weight delta ΔW(z), and a belief distribution P(z) updated continuously during the session. Each token gets a bespoke expert, generated on the fly and discarded — hence "infinite-parameter" from a fixed footprint.
Belief over point estimates. Earlier weight generators (HyperMoE, MoEGen, SHINE) generate one adapter per context — a point estimate. This paper maintains P(z) via recursive Bayesian updating:

\pi_t \propto \pi_{t-1}^{\gamma} \cdot \text{softmax}(\ell_t)

The per-token cost is K inner products, independent of session length. Topic switches shift probability mass, so the model specializes continuously rather than one-shot.

The crossover: weights win on long, noisy evidence. Results on five QA datasets:

| Dataset | Closed-book | In-context | Data-to-weights | |---------|------|------|------| | SQuAD | 20.2 | 85.3 | 51.8 | | HotpotQA | 22.1 | 58.7 | 60.4 | | 2WikiMultihopQA | 24.5 | 55.5 | 58.1 | | MuSiQue | 15.2 | 40.9 | 45.3 | | MS MARCO v2.1 | 16.8 | 33.6 | 48.0 |

Short, clean evidence favors prompts; long, noisy evidence favors compiling into weights, since a fixed-size z compresses arbitrarily long evidence at constant inference cost.

Belief accumulates across turns. In multi-turn experiments, cumulative-belief routing accuracy rises over the session while retrieval baselines (per-question retrieval, concatenated-history retrieval) stay flat or degrade — at constant rather than linear per-turn cost.
Architecture philosophy. The approach sits between frozen weights and full fine-tuning: frozen base and generator guarantee stability, only the low-dimensional P(z) moves, providing adaptation without catastrophic forgetting.

Honest limitations

1. The small generator cannot create knowledge at a large MoE's capacity — it addresses the "prompt vs. weights" question, not "big vs. small model." 2. Advantages appear only when evidence is long; short contexts still favor prompts due to generator overhead. 3. Only the discrete (categorical) belief version is implemented; continuous distributions remain future work.

Who should care

MoE researchers: a path beyond fixed expert banks.
RAG practitioners: compiling retrieved documents into weights may beat prompt stuffing on long/multi-hop QA in both cost and accuracy.
Agent designers: session history can be compiled into weights with recursive Bayesian specialization.
Inference infrastructure: per-token cost drops from O(context_length) to O(K), independent of session length.