Skip to content

Source paper

Efficient Memory Management for Large Language Model Serving with PagedAttention

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang and 6 more

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica

arXiv 2309.06180 · 202326 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper presents vLLM, a high-throughput serving system for Large Language Models (LLMs) that addresses the memory bottleneck caused by the Key-Value (KV) cache. By introducing PagedAttention, the system manages KV cache memory in a way similar to virtual memory in operating systems, allowing for near-zero fragmentation and efficient memory sharing across multiple sequences.

How it works (AI summary)

The core technique is PagedAttention, which partitions the KV cache of each request into fixed-size blocks stored in non-contiguous physical memory. A centralized scheduler and KV cache manager maintain a block table to map logical blocks to physical GPU memory, enabling dynamic allocation and copy-on-write sharing for complex decoding algorithms like beam search.