English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

CPU-ARC: Compiling LLM Reasoning into CPU Code Cuts Cost by 291x

Forum topic · ✨步子哥 · 2026-11-09

Summary

NVIDIA researchers propose CPU-ARC (Agentic Reasoning Caching), a system that 'compiles' a large language model's reasoning process into reusable, CPU-executable code. A source model (Claude Opus 4.5) solves a problem once; a builder model extracts the reasoning into parameterized 'semantic functions'; a tiny 1B-parameter model (Llama 3.2 1B) handles lightweight auxiliary tasks at inference time. On 8 benchmarks including MATH-500, AIME, GPQA Diamond, GSM8K, and MMLU-Pro, the cached CPU execution matches or exceeds source model accuracy (e.g., 93.3% vs 83.3% on AIME 2025) while cutting per-query cost by up to 291x, with a break-even point of roughly 13 queries. The cache is interpretable, robust to phrasing and parameter variations (>=90.8% accuracy), and incrementally extensible. Limitations include high one-time builder cost (12-84x a source call), applicability only to deterministic problems, and inherited reasoning errors. The paper hints at future 'semantic models' layered above neural models.

CPU-ARC: Compiling LLM Reasoning into CPU Code, 291x Cheaper

NVIDIA's research team asks a sharp question: large language models solve every problem from scratch, no matter how common. Ask an LLM how many divisors 196 has, it redoes prime factorization; ask about 144, it redoes the same derivation. The same reasoning runs thousands of times, each time burning full tokens, API fees, and GPU compute.

The Core Insight: Reasoning vs. Execution

CPU-ARC (arXiv:2611.09891) rests on one assumption: for every solvable problem, there exists a program that answers all equivalent variants of that problem correctly. Once a model has figured out the solution, that reasoning can be compiled into CPU-executable code.

Architecture: Source, Builder, Target

  • Source model (S): an expensive LLM (Claude Opus 4.5) that reasons through the problem once.
  • Builder (B): an LLM that reads the source reasoning and translates it into reusable code modules called semantic functions (e.g., load(), do(), narrate(), extract(), lm()).
  • Target (T): a small model (fine-tuned Llama 3.2 1B) doing lightweight auxiliaries like classification/extraction; real computation runs on CPU.
  • Semantic functions are parameterized: one cache entry covers an entire class of problems, not just one instance. Building a cache involves five stages: input/output isolation, decomposition, parameter identification, planning, and implementation. Builder cost is 12–84x a source call, but it is a one-time investment.

    Results: Sometimes More Accurate Than the Source Model

    | Benchmark | Source Acc. | Cache Acc. | Cost Savings | |---|---|---|---| | MATH-500 | 98.6% | 98.8% | 88.7x | | AIME 2024 | 100% | 100% | 262x | | AIME 2025 | 83.3% | 93.3% | 291x | | AIME 2026 | 90.0% | 96.7% | 281x | | GPQA Diamond | 83.8% | 83.8% | 38x | | GSM8K | 98.5% | 98.0% | 16x | | MMLU-Pro | 88.0% | 89.0% | 22.9x | | SciBench | 85.2% | 88.2% | 83x |

    The gains come from determinism: the source model sometimes reasons correctly but slips in final arithmetic, while the cache executes the same reasoning as deterministic code. Per-call cache cost is $0.0004–$0.0019 vs $0.01–$0.12 for the source model; the break-even point is around 13 calls.

    Robustness

    Tested under phrasal perturbation (rewording), parametric perturbation (new values), and combined perturbation, the cache stays above 90.8% correct on all benchmarks.

    Engineering Insights

  • Interpretable: semantic functions are auditable code — a major advantage for regulated domains.
  • Incrementally growable: new problems add functions without rebuilding existing entries; reuse increases as the cache grows.
  • Tiny target model: only a 1B-parameter model is needed at inference time; a fully compiled mode can skip model calls almost entirely.
  • Limitations

  • High one-time builder cost — only worthwhile for high-frequency, repeated queries.
  • Applies to deterministic problems, not creative or open-ended tasks.
  • Function-library generalization/merging remains unsolved.
  • Inherits source-model reasoning errors: it improves execution reliability, not reasoning quality.

The Bigger Picture

The paper envisions semantic models — a new class of fully explainable, highly efficient, continually trainable models living above neural models. LLMs become the R&D department that figures out *how* once; compiled CPU code becomes the production line that executes cheaply forever. This is knowledge externalization: like humans turning derivations into reusable formulas.

---

Paper: CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse Authors: Peter Belcak, Greg Heinrich, Saurav Muralidharan, Shizhe Diao, Pavlo Molchanov Code: not yet open-sourced (Tech Report, 28 pages)

Tags

#llm#inference-optimization#caching#cpu#cost-reduction#nvidia#reasoning#semantic-functions

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636967