Panoramic Survey of Cross-Platform Open-Source LLM Training / Inference Libraries

The Ambiguity That Must Be Split First

"A C++-based LLM library" is an ambiguous phrase. Consider three examples:

• llama.cpp — nearly everything, from tensors to scheduler to HTTP server, is C/C++; Python only does format conversion.
• PyTorch — written in C++, but you daily touch the Python layer; its C++ frontend (libtorch) exists but is rarely used.
• vLLM — 84.2% Python, 3.4% C++ — yet that 3.4% is the source of all its performance.

All three can be called "C++ libraries," with entirely different meanings. The report therefore establishes a three-layer classification:

| Layer | Definition | Criterion | Representatives | |---|---|---|---| | Layer 1 | Pure C/C++ | Compiles and runs without Python | llama.cpp, ggml, llm.c, FastLLM, bitnet.cpp, gemma.cpp | | Layer 2 | C++ core + Python shell | C++ is the product interface (stable ABI + independent releases + pure C++ path) | ONNX Runtime, OpenVINO, IREE, TensorRT-LLM, libtorch, Paddle | | Layer 3 | Python framework + C++/CUDA ops | C++ is internal implementation, untouchable by users | vLLM, SGLang, LightLLM, KTransformers |

> A critical criterion: "what language it is *written in*" and "whether developers *of that language* can *use it*" are two independent things. TensorFlow is 60% C++ yet explicitly lists tensorflow/cc as not covered by compatibility guarantees; XLA has no GitHub release and its compile options must be exported from Python as byte arrays. Conversely, ONNX Runtime and OpenVINO are true C++ stacks not because of language, but because C++ is their delivered product.

Layer 1: Pure C/C++ — Inference Reigns, Training Begs

The Landscape

| Project | License | Stars | Last Commit | Training | Status | |---|---|---|---|---|---| | llama.cpp | MIT | ~126k | 2026-08-31 | experimental llama-finetune | Very active | | ggml | MIT | 15,267 | 2026-08-30 | ggml-opt (nascent autograd) | Active | | llm.c | MIT | 30,898 | 2025-06-26 | Can pretrain (GPT-2/3) | Stalled ~14 months | | llama2.c | MIT | 20,038 | 2024-08-06 | No | Dormant ~2 years | | FastLLM | Apache-2.0 | 4,948 | 2026-08-31 | No | Daily updates | | bitnet.cpp | MIT | 40,217 | 2026-07-27 | No (1-bit) | Active | | gemma.cpp | Apache-2.0 | 7,032 | 2026-08-28 | Has VJP backward + Adam | Active | | gpt4all | MIT | 77,389 | 2025-05-27 | No | Factually dead, unarchived | | MNN | Apache-2.0 | 15,999 | 2026-08-28 | Inference-focused | Active |

(Plus llamafile, PowerInfer, ik_llama.cpp, mllm — and the dead: chatglm.cpp, qwen.cpp (archived), ctransformers, rwkv.cpp, TinyChatEngine.)

llama.cpp: Cross-Platform King via Governance, Not Engineering Brute Force

llama.cpp now has 17–18 compute backends: CUDA / Metal / Vulkan / SYCL / HIP / CANN / MUSA / OpenCL(Adreno) / BLAS / BLIS / ZenDNN / zDNN(IBM Z) / RPC / Hexagon / WebGPU / OpenVINO / ET.

The trick is the governance structure: ggml decouples hardware via a four-layer C function-pointer vtable — ggml_backend_reg_i (registration) → ggml_backend_device_i (capability query) → ggml_backend_i (execution) → ggml_backend_buffer_i (memory). The scheduler asks each device supports_op; if no, it falls down the priority list, ending with a CPU fallback.

• Minimum cost of adding a backend dropped from "modify core + pass review" to "fill four structs."
• Huawei (CANN), Moore Threads (MUSA), IBM (zDNN), Qualcomm (Hexagon), AMD (ZenDNN), Esperanto (ET) submitted their own PRs.

Eighteen backends are a product of governance structure, not engineering volume.

> But the other half of the truth: the real cost lies in the memory contract. GGML has only two static pools, no temporary-allocation API, and requires host-addressable get_base(). At FOSDEM 2025 someone documented four contract violations when writing a TTNN backend, worked around via "virtual pointer 0x1000 + true address in tensor extra." The debt of cross-platform is not the initial integration but the never-ending regression testing and operator coverage — why CANN remains Experimental.

The Training Side: A Quiet Rout

• llm.c (Karpathy): ~1000 lines of C + CUDA reproducing GPT-2/3, MPI + NCCL multi-GPU — technically sound. But the last commit was 2025-06-26 (14 months ago); it supports no Llama-family architectures; the author moved to nanochat (PyTorch).
• llama.cpp finetune: the old module was deleted by PR #8669 in 2024; the maintainer said in 2026-02: *"I can promise you that the python pile will be better than the finetune experiment… only good for finetuning extremely small models."*
• The only counterexample, gemma.cpp, ships VJP backprop and Adam — but is officially "for Gemma research," with production recommended on JAX/PyTorch.

Why? Asymmetric returns. Inference's "zero dependencies" buys coverage of tens of millions of devices. Training would require rewriting autograd + optimizer state + distributed orchestration + data pipelines to compete with PyTorch in a setting that already has GPUs and Python. The real threshold of LLM training is not forward/backward but the strategic orchestration of DDP / FSDP / TP / PP / ZeRO — combinatorial, fast-moving, user-intervened.

Mechanisms sink; strategies float. C++ ceded the middle layer.

The Mass Extinction of "Single-Model .cpp"

chatglm.cpp, qwen.cpp (archived), baichuan.cpp, ctransformers, TinyChatEngine — all stopped or archived, citing uniformly: *"in favor of llama.cpp."* After GGUF emerged as the format convergence point, the marginal return of maintaining a C++ implementation per model family went to zero.

Surviving new pure-C++ projects all compete on a different dimension:

| Project | Dimension changed | |---|---| | bitnet.cpp | Numeric format (1-bit) | | gemma.cpp | Readability (~2K LoC core) | | llamafile | Distribution form (Cosmopolitan single file, 6 OSes) | | FastLLM | Memory hierarchy (MoE experts on CUDA / CPU / NUMA / disk) |

Layer 2: C++ Core + Python Shell — The Underrated True C++ Stacks

ONNX Runtime: Undone by Its Own Name

• C/C++ is its native API layer; C#/Java/ObjC/JS/Rust all bind atop the C API.
• Execution Providers cover CUDA / TensorRT / OpenVINO / oneDNN / DirectML / QNN / NNAPI / CoreML / XNNPACK / WebGPU — unmatched coverage.
• It is the base of Windows ML and Foundry Local — an LLM runtime on every new Windows machine.

The report deems it the only C++ codebase spanning Windows/Linux/macOS/Android/iOS/Web + NVIDIA/AMD/Intel/Apple/Qualcomm.

Honest shortcomings: the GenAI C API is still "in preview"; the KV cache is not exposed (managed internally by OgaGenerator); continuous batching is still on a dev branch.

General Frameworks: High C++ Share ≠ High C++ Usability

| Framework | C++ first-class status | Verdict | |---|---|---| | PyTorch / libtorch | Complete torch::nn/optim/data/autograd | Full C++ training API, but no C++ DDP API | | PaddlePaddle (46% C++) | Inference first-class; training C++ API officially discouraged | Highest living C++ share in this layer | | MindSpore | Only custom-operator level | Shrinking into an Ascend-only stack | | TensorFlow | Low — tensorflow/cc listed as not covered | Written in C++, delivered in Python | | MXNet | — | In the Apache Attic | | OneFlow | C++ core, Python surface | Last release 2024-03-11, team disbanded |

libtorch's real position: downgraded from "deployment runtime" to "embedded tensor library." TorchScript is deprecated since PyTorch 2.10; the official path is torch.export + AOTInductor / ExecuTorch. Python defines the graph; C++ merely executes it.

Compiler Stacks: TVM, IREE, and XLA's Three Fates

• Apache TVM: runtime is pure C++, but compilation is Python-first by design; pivoting to expert-kernel infrastructure (TIRx).
• IREE: highest C++ first-class status of all — iree-compile is a standalone CLI, bare-metal with zero dependencies; but LLM ecosystem is thin and AMD-bound.
• OpenXLA: the negative exemplar — no releases, practically requires Python/JAX.

The TVM-vs-IREE "route debate" is moot in 2026: the real opposition changed from "search vs determinism" to "help experts write kernels" vs "reliably compile the graph."

Layer 3: Python Frameworks + C++/CUDA Operators

| Project | Stars | Languages | Verdict | |---|---|---|---| | vLLM | 90,567 | Py 84.2 / C++ 3.4 | C++/CUDA ops in csrc/, inseparable from Python; widest hardware coverage | | SGLang | 32,968 | Py 85.0 / C++ 2.3 | Kernels externalized into separate repos | | TensorRT-LLM | 14,511 | C++ 32.9 | True C++ runtime, --cpp_only build flag; NVIDIA-bound | | LMDeploy | 8,035 | C++ 19.6 / CUDA 10.1 | turbomind is real C++; FasterTransformer lineage; CUDA-only | | KTransformers | 19,354 | C++ 33.8 | Independent kt-kernel library | | LightLLM | 4,255 | Py 98.6 | Zero C++ — pure Triton kernels | | TGI | 10,890 | Rust 16.3 | Not C++, and archived — official migration to vLLM/SGLang | | Ollama | 179,817 | Go 70.8 / C 21.4 | "Ollama is written in C++" is false |

> Footnote: language percentages badly undercount real C++. SGLang's sgl-kernel-npu is 81.9% C++; vLLM-Ascend is C++ 52% > Python 43%. "vLLM is a Python project" describes a byproduct of repo topology, not a technical fact. Its orchestration layer is Python; its performance-critical layer (PagedAttention, quantization kernels, MoE, communication) is almost all C++/CUDA.

Edge and Mobile: C++'s Only Irreplaceable Battlefield

| Project | Platforms | Verdict | |---|---|---| | MLC-LLM (~23k) | All + Web | Only "compile-to-everything" route; WebGPU + WASM | | ExecuTorch | iOS/Android/embedded | 50KB baseline runtime; shipped in Instagram / WhatsApp / Quest 3 | | MNN | Android/iOS/server | Hexagon added; Android core .so ~800KB | | ncnn (23.7k) | 10+ platforms | Widest platform coverage, but LLM only semi-supported; int8 still on roadmap | | LiteRT-LM | Android/Chrome/Pixel Watch | Shipped in Chrome | | TNN | — | Dead (last real commit 2023-09), no archived flag |

The real edge bottleneck is memory bandwidth, not compute: iPhone A18 ~68 GB/s vs RTX 4090 ~1008 GB/s — a 15× gap explaining 4 tok/s vs 150 tok/s.

The "NPU is faster" narrative deserves a discount: NPU value is mainly perf-per-watt (40–60% power reduction). On Snapdragon 8 Gen3, QNN NPU ~32 tok/s vs llama.cpp CPU 8-thread ~18 tok/s — only ~1.8×. And NPU paths are broken: mllm's QNN decode falls back to CPU; MNN flags QNN as B-grade; ExecuTorch's Qualcomm pipeline is unfinished.

> "NPU running LLM" is currently a hybrid of prefill acceleration + decode fallback, not full offload.

A hard screening criterion: does the library have (1) a KV-cache allocator, (2) an autoregressive decode benchmark, (3) blockwise-quantized GEMM? Asking this eliminates half the candidates.

The Foundation: Operator Libraries, Quantization, Backend Abstraction, Domestic Chips

Backend Pluginization — 2026's Most Important Structural Change

Four projects in one year moved hardware backends from "compiled into core" to "runtime / out-of-tree loading":

• ONNX Runtime: Plugin EP Libraries (independently versioned releases)
• IREE: IREE_EXTERNAL_HAL_DRIVERS
• PyTorch: PrivateUse1 dispatch key (maintainer: *"PrivateUse key and device is the long term plan"*)
• TVM: src/backend/* + separate apache/tvm-ffi
• GGML: GGML_BACKEND_API_VERSION + dynamic loading

The power structure changed: integration no longer requires upstream PRs — only ABI compliance. Corollary: claims like "chip X supports PyTorch" have drastically lower technical bar; judge by operator coverage and real benchmarks.

Operators: The Moat Has Moved Up

NVIDIA's moat has shifted from "the CUDA language" to "the CuTe/CUTLASS abstraction layer" — CUTLASS pivoting to CuTe DSL, FlashAttention-4 rewritten in CuTe DSL, FlashInfer using JIT generation. Evidence: Marlin's hand-written mma loses ~37% peak throughput on Hopper; Machete required a CUTLASS rewrite to use wgmma.

FP4: Open Format ≠ Obtainable Performance

MXFP4 (OCP standard) vs NVFP4 (NVIDIA private) share element width but differ in block/scale encoding. What matters is native FP4 instructions: on Blackwell sm_120, NVFP4 boosts prefill 43–68% but decode barely moves; on RDNA4, which lacks FP4 WMMA instructions, FP4 yields bandwidth savings only, no compute gain.

Domestic (Chinese) Chips: Connected, but Only Halfway

Huawei Ascend, Cambricon, Kunlunxin, Moore Threads, MetaX all enter via PyTorch extensions (PrivateUse1) or vLLM plugins. The last mile is blocked at: (1) the distribution layer (PrivateUse1 only registers operators; no official extension path for collective communication), (2) the communication layer (fragmented HCCL/ENCCL/CNCL/MCCL, no unified ABI), and (3) the sharpest point: domestic chips have bet almost entirely on PyTorch — outside the Python ecosystem (edge, embedded, browser), they effectively do not exist.

Roundtable: Is C++ Retreating or Expanding?

The pessimist (architecture historian): the criterion is *who defines what language users write*. No C++ DDP frontend exists ("there is no C++ frontend for DDP," PyTorch maintainers, 2025-05). Four frameworks from four companies converge on the same line: C++ is guaranteed only down to the operator level. C++ retreated from "you write models with it" to "you write things others call with it."

The optimist (systems engineer): 2026's backend pluginization democratized access. Linguist statistics systematically undercount C++. Training is one-time capex; inference is perpetual opex. In serving, edge, embedded, browser, and domestic-replacement markets, C++ is not just alive but the only option. C++ lost "how to write models" but is winning "how to deliver models into people's hands."

The verdict: the pessimist speaks of *power*, the optimist of *territory* — mutually causal. Three conclusions:

1. The training orchestration layer is thoroughly Pythonized, irreversibly. Pure-C++ LLM training in 2026 is like free-soloing Everest: possible, but pointless. 2. The inference/deployment layer is alive and gained new expansion momentum via backend pluginization — integration cost dropped from "begging for PR merges" to "implementing an ABI." 3. **Judge a library by whether its C++ is a *product interface*, not whether it's *written in* C++. ONNX Runtime, OpenVINO, and IREE are true C++ stacks for exactly this reason.

Scenario-Based Selection

| # | Scenario | First choice | Avoid | |---|---|---|---| | 1 | PC local inference | llama.cpp (LM Studio / Ollama frontend) | old ggml forks | | 2 | Windows desktop · DirectML | ONNX Runtime + DirectML EP | CUDA-only solutions | | 3 | macOS native · Metal · App Store | llama.cpp (xcframework) | bundling a Python interpreter | | 4 | Android · size-sensitive | llama.cpp (CPU NEON fallback) | TNN (dead, unarchived) | | 5 | iOS · CoreML/Metal | llama.cpp (Metal) | CoreML for >3B models | | 6 | Browser · WebGPU | WebLLM / MLC-LLM | WebGL fallbacks | | 7 | GPU cluster high-throughput | vLLM (SGLang; TRT-LLM NVIDIA-only) | llama.cpp server for online concurrency | | 8 | Consumer-GPU MoE offload | llama.cpp / ktransformers | vLLM as a low-VRAM single-user setup | | 9 | Domestic chips | vendor official stacks (CANN + vLLM-Ascend) | llama.cpp directly on domestic cards | | 10 | Embedded / bare-metal | llama.cpp (pure C/C++, mature cross-compilation) | anything requiring a Python runtime |

When NOT to Use a C++ Stack

1. Server-side online inference — scheduling/batching/observability ecosystems live in Python. 2. Research and experimentation — iteration speed is the game. 3. Conversion / quantization / evaluation / data pipelines — tooling is all Python. 4. Chasing new architectures — C++ support lags weeks to months. 5. No C++ engineering capacity — memory and build costs will eat all gains.

> One-line criterion: C++'s benefit comes from deployment form (no runtime, in-process embedding, cold start, memory control), not inference speed. Speed is decided by kernels and hardware; the ecosystem is in Python.

Due-Diligence Checklist

1. Look at commits, not stars — gpt4all: 77k stars, 15 months no commits. 2. Maintainer bus factor — core committers ≤2? Acquired by a vendor? 3. Organizational migration — llama.cpp→ggml-org, PowerInfer→Tiiny-AI, llamafile→mozilla-ai. 4. Death signals without archived flags — TNN. 5. Backends per current release, not docs — ExecuTorch MPS backend deleted 2026-08; ORT removed ROCm EP in 1.23. 6. Roadmap ≠ capability — ncnn's LLM int8 is still a roadmap item. 7. License per SPDX — weights/quantization scripts separately from code. 8. ABI/API stability — versioned pure-C ABI or monthly breakage? 9. Build reproducibility — pure CMake static build, no forced Python? 10. Escape hatch (most important) — can weights migrate out (GGUF / ONNX / SafeTensors)?

Epitaph: The Departure Table

| Project | Last real commit | Death | |---|---|---| | MXNet | 2023-10-25 | Apache Attic — 12 months from graduation to retirement | | CNTK | 2023-03-11 | Archived | | Caffe | 2024-07-31 | Factually stale, unarchived | | MegEngine | 2024-10-24 | Stale 22 months | | OneFlow | 2025-12-04 | Team acquired and dispersed; founder moved to inference services, not training frameworks | | TNN | 2023-09-27 | Effectively abandoned, unarchived | | Glow / Tensor Comprehensions | — | Archived | | TGI | 2026-03-21 | Archived**; migrate to vLLM / SGLang | | qwen.cpp | 2024-12-06 | Archived | | chatglm.cpp | 2024-07-31 | Stale — *"in favor of llama.cpp"* |

Framework survival is decided by compute-ecosystem positioning, not code quality. OneFlow's distributed design was widely admired, C++ 57% — the team was acquired and dispersed anyway. MindSpore survives on Ascend (shrinking to an Ascend-only stack); Paddle survives on ERNIE + 9 domestic chips. Technically excellent but unbound: all dead.

Starting from Zero: The Correct 2026 Foundation for a Cross-Platform C++ LLM Project

• Standard: C++17 (conservative) or C++20; no C++23.
• Tensor layer: use ggml (MIT) or MNN's operator library. Never write your own operator library — the biggest pitfall.
• Weight format: GGUF as default; SafeTensors / ONNX as import only.
• Backend layering: CPU (NEON/AVX2/AVX-512) + Vulkan as universal mainlines; Metal and CUDA as platform plugins; vendor SDKs (QNN/CANN) always optional plugins, never the main path.
• Build: CMake + vendored submodules, fully static, no runtime Python discovery; expose only a pure C ABI; Swift/Kotlin/Rust bind themselves.
• Don't build yourself: quantization algorithms, tokenizer, batching scheduler — reuse the ecosystem.
• Compliance first: license inventory and domestic-environment whitelists finalized before line one of code.

Appendix: Unverified Claims (Honestly Flagged)

The original report lists 12 items it could not verify with first-hand sources, including: PowerInfer-2's missing open repository (paper arXiv:2406.06282 only); the SGLang–vLLM merger rumor (no official statement); TACO-LLM (Tencent, closed-source commercial); vendor-claim compatibility percentages for Ascend Triton and MUSA SDK (second-hand conference blogs); "WebLLM at 80% of native GPU performance"; and MNN's self-reported "8.6× prefill / 2.3× decode vs llama.cpp."

> All vendor-self-reported speedups should be treated as marketing data pending independent reproduction.

Data Notes

• All star counts, commit dates, and archive statuses come from the GitHub REST API, queried 2026-08-31; minor discrepancies are marked with "~".
• Language shares are by byte count (GitHub languages API), not lines.
• No original benchmarking was performed; all performance numbers cite sources; unverifiable items are listed as unverified.