Cross-Platform Open-Source LLM Training/Inference Libraries: A Comprehensive Survey (C++ Perspective)

Forum topic · ✨步子哥 · 2026-09-01

Summary

A Chinese tech forum post presents a panoramic survey of cross-platform open-source LLM training and inference libraries, organized by a three-layer taxonomy: pure C/C++ implementations (llama.cpp, ggml, llm.c), C++ core with Python shells (ONNX Runtime, OpenVINO, IREE, TensorRT-LLM, libtorch), and Python frameworks with C++/CUDA kernels (vLLM, SGLang, LMDeploy, KTransformers). Key findings: llama.cpp's 17-18 hardware backends result from a governance structure of four C function-pointer vtables rather than raw engineering effort; training in pure C++ has quietly collapsed (llm.c stalled 14 months, llama.cpp's finetune is experimental) because gains are asymmetric versus PyTorch; backend plugin-ization in 2026 (ONNX Runtime plugin EPs, PyTorch PrivateUse1, ggml dynamic loading) shifted ecosystem power toward out-of-tree vendors, including Chinese chip makers (Huawei Ascend, Moore Threads). Edge/mobile is C++'s irreplaceable battleground, constrained by memory bandwidth and incomplete NPU decode paths. The report concludes C++ lost the training orchestration layer irreversibly to Python but is expanding in inference deployment, and offers a ten-scenario selection matrix, a ten-item due-diligence checklist, an obituary of dead frameworks (MXNet, TGI, OneFlow), and a build-from-scratch stack recommendation centered on ggml, GGUF, CMake, and a pure C ABI.

Key points

"A C++ LLM library" is ambiguous. The report defines three layers: (1) pure C/C++ implementations (llama.cpp, ggml, llm.c, FastLLM, bitnet.cpp, gemma.cpp), (2) C++ core with Python shell where C++ is the *product interface* (ONNX Runtime, OpenVINO, IREE, TensorRT-LLM, libtorch, Paddle), and (3) Python frameworks with C++/CUDA kernels invisible to users (vLLM, SGLang, LightLLM, KTransformers). Being *written* in a language and being *usable* by its developers are independent things—TensorFlow is 60% C++ yet lists tensorflow/cc outside compatibility guarantees.
llama.cpp's 18 backends are a governance artifact, not an engineering artifact. ggml decouples hardware via four layers of C function-pointer vtables (registry → device → backend → buffer). Adding a backend costs filling four structs, so Huawei (CANN), Moore Threads (MUSA), IBM (zDNN), Qualcomm (Hexagon) submit their own PRs. The real debt is the memory contract and endless regression testing, which is why CANN remains Experimental.
Training in pure C++ has quietly collapsed. llm.c is stalled ~14 months; llama.cpp's old finetune was deleted (PR #8669) and the current llama-finetune is only for tiny models; gemma.cpp's VJP+Adam is self-labeled research-only. The barrier isn't forward/backward but strategic orchestration (DDP/FSDP/TP/PP/ZeRO)—Python owns that layer irreversibly. Single-model .cpp projects (chatglm.cpp, qwen.cpp, ctransformers) went extinct once GGUF converged formats; survivors compete on a different dimension (bitnet.cpp: 1-bit; llamafile: single-file; FastLLM: memory tiering).
Language statistics understate real C++ content. vLLM shows only 3.4% C++, but its kernels live in csrc/ and split repos (sgl-kernel-npu is 81.9% C++). vLLM's orchestration is Python; its performance-critical layer is C++/CUDA. Ollama is Go, not C++.
ONNX Runtime is the only C++ stack covering Windows/Linux/macOS/Android/iOS/Web and NVIDIA/AMD/Intel/Apple/Qualcomm; it underpins Windows ML. Shortcomings: GenAI C API in preview, KV cache not exposed, no continuous batching on mainline yet. TVM vs IREE no longer conflict—TVM pivots to expert kernel infrastructure (TIRx), IREE to deterministic pipelines with a bare-metal C runtime.
2026's key structural change: backend plugin-ization. ONNX Runtime plugin EPs, IREE external HAL drivers, PyTorch PrivateUse1, TVM FFI, ggml dynamic loading all move integration from "merge a PR" to "implement an ABI." This matters most for Chinese accelerators (Ascend, Cambricon, Kunlunxin, MUSA, MetaX)—but they almost all bet on PyTorch and essentially vanish outside Python ecosystems. Claim of "chip supports PyTorch/ORT" must now be judged by operator coverage and real performance.
Edge is C++'s irreplaceable battlefield, bottlenecked by memory bandwidth (A18 ≈68 GB/s vs RTX 4090 ≈1008 GB/s), not compute. "NPU is faster" needs discounting: gains are per-watt (40–60% power cut), ~1.8× throughput, and NPU paths are hybrid (prefill accelerated, decode falls back to CPU). Hard screening test: does the library have a KV-cache allocator, autoregressive decode benchmarks, and blockwise-quantized GEMM? This filters half the candidates (ncnn only recently added KV-cache support; int8 LLM still on its roadmap).
The NVIDIA moat moved up to CuTe/CUTLASS abstraction: FlashAttention-4 rewritten in CuTe DSL, FlashInfer uses JIT generation; hand-written mma (Marlin) loses ~37% of Hopper peak. FP4: MXFP4 is an open OCP standard, NVFP4 proprietary—Blackwell gets native FP4 compute gains in prefill, RDNA4 only bandwidth savings. Same GGUF file, different payoffs.

Scenario selection matrix (highlights)

| Scenario | First choice | |---|---| | PC local | llama.cpp (via LM Studio/Ollama frontend) | | Windows/DirectML | ONNX Runtime + DirectML EP | | macOS native (App Store) | llama.cpp xcframework, pure C API | | Browser/WebGPU | WebLLM / MLC-LLM | | GPU cluster serving | vLLM; SGLang alternative; TRT-LLM if NVIDIA-only | | Consumer-GPU MoE offload | llama.cpp or KTransformers | | Chinese domestic silicon | vendor stacks (CANN + vLLM-Ascend etc.) | | Embedded/bare-metal | llama.cpp; MNN/ncnn for small models |

Avoid C++ stacks for: server-side online serving, research experimentation, tooling pipelines, chasing new architectures, or teams without C++ engineering capability. C++'s payoff comes from deployment shape (no runtime, in-process embedding, cold start, memory control)—not inference speed.

Due diligence and graveyard

Ten checks include: trust commits not stars (gpt4all: 77k stars, 15 months dead), watch bus factor, verify backends against current releases (ExecuTorch's MPS backend deleted 2026-08; ORT removed ROCm EP in 1.23), roadmap ≠ capability, and always secure an escape hatch for weight formats (GGUF/ONNX/SafeTensors).

The obituary table records the complete extinction of native C++ DL frameworks: MXNet (Apache Attic, retired 12 months after graduating), CNTK, Caffe, MegEngine, OneFlow (team acquired and dispersed; pivoted to inference serving), TNN, Glow, TGI (archived, recommending vLLM/SGLang). Framework survival is decided by compute ecosystem position, not code quality.

Building from scratch in 2026

C++17/20; use ggml or MNN for the tensor layer (never write your own operator library); GGUF as default weight format; CPU(NEON/AVX)+Vulkan as universal backends, Metal/CUDA as platform plugins, vendor SDKs as optional plugins only; CMake, full static linking, pure C ABI externally; don't build quantization, tokenizer, or batching yourself.

Caveats

The report notes data was pulled live from the GitHub REST API (2026-08-31), and candidly lists unverified items, including PowerInfer-2's missing repo, rumored SGLang/vLLM merger (no official confirmation), vendor-claimed Ascend/MUSA compatibility numbers, and MNN's self-reported speedups—all flagged as requiring independent reproduction.