Robust Suffix-Aware KV Cache Indexing
Implement a rolling hash indexing mechanism within the Token Processor to enable robust KV cache retrieval despite sliding-window context truncation.
Source paper
Yuhan Liu, Yihua Cheng, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Rui Zhang, Kuntai Du, Junchen Jiang
Generated by AI. The paper’s authors did not write these.
Implement a rolling hash indexing mechanism within the Token Processor to enable robust KV cache retrieval despite sliding-window context truncation.
Develop a real-time cost-model scheduler inside the KV Connector that arbitrates between network KV cache loading and GPU re-computation based on bandwidth and token length.
Extend storage APIs to support sparse KV retrieval by incorporating accumulated attention score metadata, transferring only high-value cache pages to minimize bandwidth.
This paper introduces LMCache, a middleware layer designed to decouple Key-Value (KV) cache management from LLM inference engines like vLLM and SGLang. By implementing efficient, chunk-based data transfer protocols and a standardized connector interface, LMCache enables high-performance cross-query context sharing and cross-engine prefill-decode disaggregation, significantly reducing latency and improving throughput in enterprise settings.
LMCache functions as a distributed KV cache store that sits between inference engines and storage backends (memory, disk, S3). It replaces inefficient page-wise transfers with optimized 'chunk' transfers (default 256 tokens), utilizes zero-copy operations, and pipelines compute with I/O. It exposes a 'KV connector' API to inference engines for retrieving cached states and a 'Controller' for global cache orchestration.