CPU-ARC: Compiling LLM Reasoning into CPU Code, 291x Cheaper
NVIDIA's research team asks a sharp question: large language models solve every problem from scratch, no matter how common. Ask an LLM how many divisors 196 has, it redoes prime factorization; ask about 144, it redoes the same derivation. The same reasoning runs thousands of times, each time burning full tokens, API fees, and GPU compute.
The Core Insight: Reasoning vs. Execution
CPU-ARC (arXiv:2611.09891) rests on one assumption: for every solvable problem, there exists a program that answers all equivalent variants of that problem correctly. Once a model has figured out the solution, that reasoning can be compiled into CPU-executable code.
Architecture: Source, Builder, Target
- Source model (S): an expensive LLM (Claude Opus 4.5) that reasons through the problem once.
- Builder (B): an LLM that reads the source reasoning and translates it into reusable code modules called semantic functions (e.g.,
load(),do(),narrate(),extract(),lm()). - Target (T): a small model (fine-tuned Llama 3.2 1B) doing lightweight auxiliaries like classification/extraction; real computation runs on CPU.
- Interpretable: semantic functions are auditable code — a major advantage for regulated domains.
- Incrementally growable: new problems add functions without rebuilding existing entries; reuse increases as the cache grows.
- Tiny target model: only a 1B-parameter model is needed at inference time; a fully compiled mode can skip model calls almost entirely.
- High one-time builder cost — only worthwhile for high-frequency, repeated queries.
- Applies to deterministic problems, not creative or open-ended tasks.
- Function-library generalization/merging remains unsolved.
- Inherits source-model reasoning errors: it improves execution reliability, not reasoning quality.
Semantic functions are parameterized: one cache entry covers an entire class of problems, not just one instance. Building a cache involves five stages: input/output isolation, decomposition, parameter identification, planning, and implementation. Builder cost is 12–84x a source call, but it is a one-time investment.
Results: Sometimes More Accurate Than the Source Model
| Benchmark | Source Acc. | Cache Acc. | Cost Savings | |---|---|---|---| | MATH-500 | 98.6% | 98.8% | 88.7x | | AIME 2024 | 100% | 100% | 262x | | AIME 2025 | 83.3% | 93.3% | 291x | | AIME 2026 | 90.0% | 96.7% | 281x | | GPQA Diamond | 83.8% | 83.8% | 38x | | GSM8K | 98.5% | 98.0% | 16x | | MMLU-Pro | 88.0% | 89.0% | 22.9x | | SciBench | 85.2% | 88.2% | 83x |
The gains come from determinism: the source model sometimes reasons correctly but slips in final arithmetic, while the cache executes the same reasoning as deterministic code. Per-call cache cost is $0.0004–$0.0019 vs $0.01–$0.12 for the source model; the break-even point is around 13 calls.
Robustness
Tested under phrasal perturbation (rewording), parametric perturbation (new values), and combined perturbation, the cache stays above 90.8% correct on all benchmarks.
Engineering Insights
Limitations
The Bigger Picture
The paper envisions semantic models — a new class of fully explainable, highly efficient, continually trainable models living above neural models. LLMs become the R&D department that figures out *how* once; compiled CPU code becomes the production line that executes cheaply forever. This is knowledge externalization: like humans turning derivations into reusable formulas.
---
Paper: CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse Authors: Peter Belcak, Greg Heinrich, Saurav Muralidharan, Shizhe Diao, Pavlo Molchanov Code: not yet open-sourced (Tech Report, 28 pages)