The rapid shift of language models from research artifacts to production-critical systems has forced a deep re-evaluation of how inference workloads are architected. Early optimization efforts focused almost exclusively on training, where performance is dominated by dense matrix multiplications and raw FLOPs. In contrast, large-scale inference has revealed a different truth: serving language models efficiently is not compute-bound, but memory-bound. The dominant constraint is no longer GPU arithmetic throughput, but how effectively memory bandwith.
During inference, transformer-based models generate tokens sequentially. For each new token, the model must attend to all previously generated tokens to preserve context. To avoid recomputing attention inputs at every step, a process that would scale quadratically with sequence length, serving engines cache intermediate key and value tensors (aka KV cache) in GPU memory. This KV cache grows linearly with both sequence length and model depth, quickly becoming the largest consumer of memory. For modern models with long context windows, the KV cache high likelly exceeds the size of the model weights themselves.
Traditional inference stacks inherited their memory management strategies from training frameworks, where static shapes and predictable lifetimes dominate. Applied to inference, these assumptions break down. Legacy systems typically pre-allocate KV cache memory for the maximum possible sequence length, regardless of how short a prompt or response actually is. This leads to severe over-reservation, compounded by internal fragmentation from fixed-size slabs and external fragmentation caused by the requirement that each sequence’s KV cache occupy a contiguous block of memory. The result is strikingly poor utilization: in many production setups, only 30–40% of available GPU memory is actually used for active tokens – 60-70% waste. From a cost perspective, this means that a significant portion of the investment in accelerators such as those from NVIDIA delivers no effective throughput.
PagedAttention emerged as a response to this systemic inefficiency, applying a principle from operating systems, virtual memory paging, to the KV cache. Instead of allocating one large contiguous buffer per sequence, PagedAttention divides the KV cache into small, fixed-size blocks, each holding the keys and values for a limited number of tokens. Logical token order is decoupled from physical memory layout via a block table that maps sequence positions to arbitrary memory blocks.
This seemingly simple abstraction has far-reaching consequences. Memory is allocated on demand as tokens are generated, eliminating over-reservation entirely. Because blocks are uniform and interchangeable, external fragmentation disappears: any free block can be reused by any sequence. Internal fragmentation is limited to the final, partially filled block of a sequence, reducing waste to a handful of tokens rather than thousands. In practice, systems built on PagedAttention reduce KV cache waste from as high as 80% to well under 5%, fundamentally changing the LLM serving code. The vLLM inference engine builds directly on PagedAttention and extends it into a full production-grade architecture. By treating KV blocks as reference-counted objects, vLLM enables efficient memory sharing across sequences with common prefixes. Parallel sampling and beam search benefit immediately: multiple candidate continuations can share the same prompt cache, with copy-on-write semantics ensuring correctness when paths diverge. At scale, this can cut memory usage by more than half for complex decoding workloads.