PagedAttention: How Virtual Memory Paging Solved LLM KV Cache Waste

Forum topic · ✨步子哥 · 2026-07-20

Summary

PagedAttention is a memory management technique introduced by the vLLM team to address the inefficiency of KV cache allocation in transformer-based large language models. Traditional systems required contiguous GPU memory blocks for each sequence, causing severe fragmentation and wasting 60-80% of GPU memory when sequence lengths are unpredictable. Inspired by operating system virtual memory paging, PagedAttention divides KV cache into fixed-size blocks (e.g., 16 tokens each) and uses a block table to map logical to physical locations, eliminating the need for contiguous allocation. This approach raises GPU utilization to near 100% and enables efficient memory sharing across parallel sampling requests through copy-on-write semantics. The technique preserves the transformer math unchanged, demonstrating that classical systems engineering can solve modern deep learning infrastructure bottlenecks elegantly.

Key points

KV cache problem in LLMs: Transformer models store Key/Value pairs for every processed token to maintain context. Because output length is unknown ahead of time, traditional systems must pre-allocate contiguous memory per sequence. Over-allocating wastes memory; under-allocating triggers expensive reallocation. Fragmentation can waste 60–80% of GPU memory.
Core idea — borrow OS virtual memory: The vLLM team adapted classical paging into the GPU. Memory is divided into fixed-size blocks (commonly 16 tokens per block). A per-sequence block table records the physical location of each block, so the GPU no longer needs contiguous space.
Results: Near-100% GPU utilization, roughly 2× serving throughput on the same hardware, and elimination of internal/external fragmentation.
Parallel sampling and copy-on-write: When multiple beams or samples share a common prompt prefix, they can share the same physical KV blocks. New blocks are allocated only when the sequences diverge — the same *copy-on-write* semantics used by operating systems.
Engineering insight: PagedAttention does not change transformer math. It is a memory-management technique that reframes LLM serving as a systems problem with a 40-year-old solution.

Why it matters

PagedAttention turns the KV cache from a rigid, contiguous slab into a flexible collection of small pages. The block table acts as a lightweight directory, letting the attention kernel gather scattered blocks on the fly. This makes high-throughput LLM serving practical on commodity GPUs and underpins the vLLM inference engine.

Reference

Paper: Kwon et al., *Efficient Memory Management for Large Language Model Serving with PagedAttention* (SOSP 2023). Project: https://blog.vllm.ai/

PagedAttention: How Virtual Memory Paging Solved LLM KV Cache Waste

Summary

Key points

Why it matters

Reference

Tags