Cross-Platform Open-Source LLM Training and Inference Libraries: A Full Landscape Report

Forum topic · ✨步子哥 · 2026-09-01

Summary

A comprehensive landscape report on cross-platform open-source C/C++ LLM libraries, proposing a three-tier taxonomy: pure C/C++ implementations (llama.cpp, ggml, llm.c), C++ cores with Python shells (ONNX Runtime, OpenVINO, IREE, TensorRT-LLM, libtorch), and Python frameworks with C++/CUDA kernels (vLLM, SGLang). Key findings: llama.cpp's 18 backend portability stems from a four-layer C vtable governance structure rather than raw engineering effort; C++ has lost LLM training entirely (no DDP C++ API, llm.c stalled, gemma.cpp research-only); single-model .cpp projects went extinct after GGUF convergence; 2026 marked a structural shift to runtime-plugin hardware backends (ONNX Runtime plugin EPs, PyTorch PrivateUse1, GGML backend API v2), lowering vendor integration costs. The report includes a graveyard of dead frameworks (MXNet, OneFlow, TGI), on-device analysis (memory bandwidth as the true bottleneck, NPU partial offload reality), FP4 quantization caveats, ten scenario-based recommendations, a ten-item due-diligence checklist, and honest notes on unverified vendor benchmarks.

Key points

This is a structured summary of a long Chinese forum report surveying cross-platform open-source C/C++ libraries for LLM training and inference. All statistics were pulled from the GitHub REST API on 2026-08-31.

1. A three-tier taxonomy is required

Tier 1 — Pure C/C++: compiles and runs without Python. Examples: llama.cpp, ggml, llm.c, FastLLM, bitnet.cpp, gemma.cpp.
Tier 2 — C++ core as product interface: stable ABI, independent releases, Python-free path. Examples: ONNX Runtime, OpenVINO, IREE, TensorRT-LLM, libtorch, Paddle.
Tier 3 — Python framework with C++/CUDA kernels: C++ is internal implementation. Examples: vLLM, SGLang, LightLLM, KTransformers.

Critical distinction: what a library is *written in* (TensorFlow is 60% C++ but exposes Python; tensorflow/cc is explicitly outside compatibility guarantees; XLA has no GitHub release) versus what it *delivers* to C++ developers.

2. llama.cpp: portability through governance, not engineering

17–18 backends (CUDA, Metal, Vulkan, SYCL, HIP, CANN, MUSA, OpenCL, Hexagon, WebGPU, OpenVINO, etc.) built on a four-layer C function-pointer vtable: ggml_backend_reg_i → ggml_backend_device_i → ggml_backend_i → ggml_backend_buffer_i, with CPU fallback.
Vendors (Huawei, Moore Threads, IBM, Qualcomm, AMD, Esperanto) submit their own backend PRs; the cost of adding a backend is filling four structs.
The real cost lies in the memory contract: static weight/scratch pools, no temp-allocation API, host-addressable get_base() requirement. FOSDEM 2025 documented four violations when writing a TTNN backend. Regression testing and operator coverage — not initial integration — are the never-ending debt (CANN backend remains Experimental).

3. Training: a quiet collapse

llm.c (Karpathy): technically sound GPT-2/3 pretraining in ~1000 lines of C+CUDA with MPI+NCCL, but stalled since 2025-06-26; no Llama-family support; author moved to PyTorch-based nanochat.
llama.cpp: old finetune module deleted (PR #8669); current llama-finetune explicitly dismissed by maintainers as only good for extremely small models.
gemma.cpp is the sole counterexample (VJP backward + Adam), but is officially research-only.
Root cause: asymmetric payoff. Inference-side zero-dependency wins millions of devices; training would require rebuilding autograd, optimizers, and DDP/FSDP/TP/PP/ZeRO orchestration against an entrenched Python ecosystem. PyTorch maintainers confirmed: *"there is no C++ frontend for DDP."*
Single-model .cpp projects (chatglm.cpp, qwen.cpp, ctransformers, etc.) all died "in favor of llama.cpp" after GGUF became the format convergence point. Surviving new entrants compete on a different axis: bitnet.cpp (1-bit), gemma.cpp (readability), llamafile (single-file distribution), FastLLM (memory hierarchy for MoE).

4. Tier 2: underrated true C++ stacks

ONNX Runtime (21.7k stars): C API is the native layer; unmatched EP coverage (CUDA, TensorRT, DirectML, QNN, CoreML, WebGPU, ...); powers Windows ML and Foundry Local. Weak spots: GenAI C API in preview, KV cache not exposed, continuous batching still on dev branch, ROCm EP removed in 1.23.
libtorch has been downgraded from deployment runtime to embedded tensor library: TorchScript deprecated in favor of torch.export + AOTInductor/ExecuTorch; no C++ DDP API.
IREE offers the highest C++ first-class treatment (bare-metal, standalone iree-compile CLI); TVM pivoted to expert kernel infrastructure (TIRx). The old TVM-vs-IREE debate is moot — different tracks now.
OpenVINO GenAI: highest LLM feature density (EAGLE-3 speculative decoding, MoE offload to disk).

5. Tier 3: language ratios understate real C++ content

vLLM shows only 3.4% C++, but its performance-critical kernels live in csrc/ — and derivative repos carry the rest: sgl-kernel-npu is 81.9% C++, vLLM-Ascend is 52% C++. "vLLM is a Python project" describes repository topology, not technical reality; its orchestration layer is Python, its bottleneck layer is C++/CUDA.
LMDeploy turbomind: overrated on cross-platform (CUDA only) but underrated on lineage — the only living production-grade C++ serving engine descending from FasterTransformer.
TGI is archived (Rust, not C++); "Ollama is written in C++" is false (Go 70.8%).

6. On-device: C++'s irreplaceable battlefield

The real bottleneck is memory bandwidth (iPhone A18 ~68 GB/s vs RTX 4090 ~1008 GB/s, a 15× gap), not compute.
"NPU is faster" needs discounting: NPU value is per-watt (40–60% power reduction); QNN NPU is only ~1.8× faster than 8-thread CPU on Snapdragon 8 Gen3. NPU paths are incomplete everywhere — typically prefill acceleration with CPU decode fallback.
Hard filter for candidates: does it have a KV-cache allocator, autoregressive decode benchmarks, and quantized GEMM? MLC-LLM (compiles to WebGPU/WASM), ExecuTorch (shipped in Instagram/WhatsApp/Quest 3), and MNN pass; ncnn's LLM support is only partial (int8 still on roadmap). Note: ExecuTorch's MPS backend was removed in 2026-08; TNN is dead without an archive flag.

7. Infrastructure shifts in 2026

Backend pluginization: ONNX Runtime plugin EPs now version independently, IREE external HAL drivers, PyTorch PrivateUse1 dispatch key, GGML backend API v2 with dynamic loading. Integration no longer requires upstream PRs — only ABI compliance. Consequently, claims like "chip X supports PyTorch/ORT" carry far less weight; judge by operator coverage and measured performance.
NVIDIA's moat moved up from CUDA to the CuTe/CUTLASS abstraction: FlashAttention-4 rewritten in CuTe DSL, FlashInfer using JIT codegen; hand-written mma kernels (Marlin) lose ~37% peak on Hopper.
FP4: MXFP4 (OCP standard) vs NVFP4 (NVIDIA) share element format but differ in block scaling. Performance depends on native instructions: Blackwell gets prefill gains of 43–68% (decode unchanged); RDNA4 has no FP4 WMMA, so FP4 yields only bandwidth savings.
Chinese chips: integration is half-finished — proprietary collectives (HCCL/CNCL/MCCL) lack a unified ABI, PrivateUse1 leaves distribution and benchmark-timer extensions unsolved, and outside PyTorch (edge, embedded, browser) these chips effectively don't exist. Only MUSA (Stable) and CANN (Experimental) have llama.cpp backends.

8. Verdict

Training orchestration is irreversibly Python-ized (asymmetric payoff, not technical infeasibility). Inference/deployment C++ is not only alive but gained new expansion momentum through backend pluginization. Judge a library by whether its C++ is a product interface (stable ABI, independent releases), not by what it's written in.

9. Ten scenario recommendations (condensed)

1. PC local: llama.cpp (front-end LM Studio/Ollama) 2. Windows embedded/DirectML: ONNX Runtime + DirectML EP 3. macOS native App Store: llama.cpp xcframework 4. Android size-sensitive: llama.cpp + MNN backup; avoid TNN 5. iOS: llama.cpp Metal; CoreML only ≤3B 6. Browser: WebLLM/MLC-LLM 7. GPU cluster serving: vLLM (SGLang, TRT-LLM for NVIDIA-only) 8. Consumer-GPU MoE offload: llama.cpp / KTransformers 9. Chinese-accelerator compliance: vendor official stacks (CANN + vLLM-Ascend) 10. Embedded/bare-metal: llama.cpp; MNN/ncnn for small models

When not to use C++: server-side online inference, research iteration, tooling pipelines, fast-follow of new architectures, or teams without C++ engineering capability. C++ benefits come from deployment form (no runtime, in-process, cold-start, memory control) — not inference speed.

10. Due-diligence checklist

Check commits not stars (gpt4all: 77k stars, 15 months inactive); bus factor; org migrations (llama.cpp→ggml-org, PowerInfer→Tiiny-AI); death without archive flag (TNN); backend status per release not docs (MPS removed, ROCm EP removed); roadmap ≠ capability; SPDX-license audit; ABI stability; reproducible builds; and migration escape hatches (GGUF/ONNX/SafeTensors).

11. Framework graveyard

MXNet (Apache Attic 12 months after graduation), CNTK, Caffe, MegEngine (22 months stalled), OneFlow (team acquired and dispersed; founder's new company went public pursuing inference serving, not training frameworks), TNN, Glow, Tensor Comprehensions, TGI, qwen.cpp, chatglm.cpp. Framework survival is decided by compute-ecosystem positioning, not code quality — technically excellent but unbound projects all died.

12. Greenfield advice and unverified claims

If starting from scratch in 2026: C++17/20; use ggml or MNN operators (never build your own); GGUF default format; Vulkan+CPU as universal mainlines, Metal/CUDA as platform plugins, vendor SDKs as optional plugins; CMake vendored static builds exposing only a pure C ABI; reuse quantization, tokenizers, and batching schedulers.

The report's appendix honestly lists unverified items, including PowerInfer-2's missing repo, SGLang/vLLM merger rumors, vendor-claimed Ascend/MUSA compatibility figures, and MNN's self-reported 8.6× prefill speedup — all flagged as requiring independent reproduction.

> Any vendor-self-benchmarked speedup should be treated as marketing data until independently reproduced.