Infinite-Parameter LLMs: Compiling Knowledge into Weights Instead of Prompts

This forum post discusses an arXiv paper (2609.18842) that proposes compiling runtime data into model weights instead of placing it in the prompt — challenging the decade-old assumption that LLM weights should stay frozen after training.

The problem with context-only memory

Every modern LLM (GPT-5, Claude Opus 5, Gemini 3) re-reads the entire conversation history through its transformer layers on every inference. All runtime data — RAG documents, few-shot examples, system prompts — lives only in the context window. This has two fundamental costs:

• Compute grows linearly with context length. Every generated token re-attends over the full context.
• Knowledge vanishes when the session ends. Everything the user taught the model lives in the prompt, never in the weights.

Existing remedies (test-time training, fast weights) either produce point estimates, suffer catastrophic forgetting, or are too expensive.

From MoE to generated experts

MoE models like DeepSeek-V3 (671B total, 37B active per token) already make weights input-dependent, but their expert bank is fixed after training. The paper's move: don't store an expert bank — generate experts from live data. Three components replace the standard FFN layer:

1. Frozen base FFN — always active. 2. Generator — a small network mapping a latent code z to a low-rank weight delta ΔW(z). 3. Belief over latent code — a distribution P(z), updated continuously through the session.

There is no stored expert bank; each token's expert is compiled on the fly and discarded. "Infinite-parameter" means a fixed-size footprint can generate unlimited effective weight combinations.

Belief, not point estimates

Unlike prior weight generators (HyperMoE, MoEGen, SHINE) that produce one adapter per session, the key innovation is maintaining a distribution P(z) rather than a single z. It updates via recursive Bayes:

\[\pi_t \propto \pi_{t-1}^{\gamma} \cdot \text{softmax}(\ell_t)\]

where ℓ_t is evidence from token t and γ ∈ [0,1] controls forgetting speed. Cost per token is K inner products (K = candidate codes), independent of session length — so the model keeps specializing mid-conversation (e.g., when the topic switches) at flat cost.

When weights beat prompts

Benchmark results across five QA datasets (from easy to hard evidence):

| Dataset | Evidence | Closed-book | In-context | Data-to-weights | |---|---|---|---|---| | SQuAD | 1 short passage | 20.2 | 85.3 | 51.8 | | HotpotQA | 2-hop + distractors | 22.1 | 58.7 | 60.4 | | 2WikiMultihopQA | multi-hop | 24.5 | 55.5 | 58.1 | | MuSiQue | hard multi-hop | 15.2 | 40.9 | 45.3 | | MS MARCO v2.1 | 10 noisy passages | 16.8 | 33.6 | 48.0 |

The pattern is clear: prompting wins on short, clean evidence; compiling into weights wins on long, noisy evidence, where compiled knowledge stays a fixed-size z regardless of source length.

Belief accumulation beats retrieval baselines

In multi-turn experiments over a fixed knowledge pool, cumulative belief routing achieved rising accuracy on context-dependent questions, while per-question retrieval stayed flat and history-concatenation retrieval degraded — all at constant per-turn cost versus linear growth for concatenated queries.

Honest limitations

• The generator cannot create knowledge beyond its own capacity — it handles runtime data, not the role of a large MoE's stored knowledge.
• The advantage only appears with long evidence; short contexts are still better served by prompting.
• Only the discrete (categorical) belief version is implemented; the continuous version is future work.

Who should care

• MoE researchers: experts can be generated from data, with accumulating beliefs — a new design space beyond fixed expert banks.
• RAG practitioners: compiling retrieved documents into weights may beat stuffing them into prompts for long, multi-hop tasks.
• Agent designers: belief accumulation offers a third path beyond growing contexts or external memory.
• Inference infrastructure: cost changes from O(context_length) to O(K) per token, independent of session length.

Why it matters

The paper redefines where learning happens: not just pretraining and fine-tuning, but at deployment — via cheap, reversible Bayesian belief updates instead of gradient descent. It is a new middle ground between frozen weights and full fine-tuning: dynamic enough to adapt to live data, stable enough to avoid catastrophic forgetting. The architecture resembles the brain — a stable cortex (frozen base FFN) with a hippocampus-like system (generator + belief) re-encoding new experience.

Paper links

• Paper: Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
• arXiv HTML: https://arxiv.org/html/2609.18842v1