This forum post explains why large language model (LLM) inference is bottlenecked not by compute but by wasteful KV cache memory allocation, and how vLLM's PagedAttention—which borrows OS-style virtual memory paging to manage KV cache in fixed-size blocks with a block table—reduced memory fragmentation below 4% and boosted throughput 2–4x. It compares five major inference engines: vLLM (general-purpose king, easy pip-based deployment, broad hardware support), NVIDIA TensorRT-LLM (15–35% higher throughput on fixed NVIDIA clusters but painful builds and weak support for new model architectures), SGLang (RadixAttention prefix-tree caching that excels for multi-agent and structured JSON generation), Hugging Face TGI (steady mid-tier option), and llama.cpp (pure C++ for local and edge use). The post derives the KV cache memory formula showing linear scaling with concurrency and context length, explains continuous (in-flight) batching, and provides a decision guide based on hardware, workload, and team priorities.
vLLM vs. TensorRT-LLM vs. SGLang vs. TGI vs. llama.cpp: How PagedAttention Solved the VRAM Black Hole
Summary
This forum post explains why large language model (LLM) inference is bottlenecked not by compute but by wasteful KV cache memory allocation, and how vLLM's PagedAttention—which borrows OS-style virtual memory paging to manage KV cache in fixed-size blocks with a block table—reduced memory fragmentation below 4% and boosted throughput 2–4x. It compares five major inference engines: vLLM (general-purpose king, easy pip-based deployment, broad hardware support), NVIDIA TensorRT-LLM (15–35% higher throughput on fixed NVIDIA clusters but painful builds and weak support for new model architectures), SGLang (RadixAttention prefix-tree caching that excels for multi-agent and structured JSON generation), Hugging Face TGI (steady mid-tier option), and llama.cpp (pure C++ for local and edge use). The post derives the KV cache memory formula showing linear scaling with concurrency and context length, explains continuous (in-flight) batching, and provides a decision guide based on hardware, workload, and team priorities.
This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178634546