Dynamic Network-Compute Latency Arbitrator
Develop a real-time cost-model scheduler inside the KV Connector that arbitrates between network KV cache loading and GPU re-computation based on bandwidth and token length.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 8.7 ('Sensitivity Study') and Figure 15 demonstrate that at lower network bandwidths (e.g., 32 Gbps), loading the KV cache is actually slower than re-computing the prefill for context lengths under 256K tokens.
“Thus, adaptive decisions between KV cache loading and prefilling need to be made when KV cache resides in a remote storage server.”
Limitation
The current system lacks a real-time decision mechanism to choose between 'load' and 'compute'. It blindly loads from cache if available, which incurs a performance penalty when network bandwidth is constrained or the model's arithmetic intensity is low.
Proposed Solution
Develop a cost-model-based scheduler inside the KV Connector. This module would monitor real-time network throughput and GPU compute utilization to calculate `Predicted_Load_Time` vs `Predicted_Prefill_Time`. It would dynamically route the request to either fetch the KV cache or trigger a re-computation, ensuring the system strictly adheres to the lowest latency path.
Why This Matters
This optimization ensures LMCache never degrades performance below the baseline, making it robust for edge environments or shared cloud networks with variable bandwidth.
How to Get Started
How to test it: We hypothesize that a KV-connector cost model comparing predicted remote load latency against predicted prefill latency will achieve lower mean and p95 TTFT than both always-loading LMCache and always-prefilling basic vLLM under constrained or fluctuating network bandwidth. Evaluating on LongBench TriviaQA and 8K-512K context sweeps across 15-128 Gbps links, we should stop if estimation errors yield >5% worse p95 TTFT than the better static policy or scheduling overhead exceeds 5 ms per query.
- Profile network transfer times versus prefill computation latencies across various GPU types and network bandwidths.
- Implement a dynamic cost model that calculates predicted load time versus predicted prefill time.
- Embed the arbitrator scheduler inside the KV Connector request routing pipeline.
- Benchmark end-to-end request latency under simulated fluctuating network conditions using vLLM and LMCache.
Code linked to this paper on Hugging Face: github.com/LMCache/LMCache
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the provided literature addresses a scheduler inside a KV connector that dynamically weighs network throughput and GPU compute utilization to choose between fetching a KV cache or re-computing.
We searched OpenAlex for “Develop a real-time cost-model scheduler inside the KV Connector that arbitrates between network KV cache loading and GPU re-computation based on bandwidth and token length.” and “Dynamic Network-Compute Latency Arbitrator” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM InferenceYuhan Liu, Yihua Cheng, Jiayi Yao +8 more · 2025
This paper introduces LMCache, a middleware layer designed to decouple Key-Value (KV) cache management from LLM inference engines like vLLM and SGLang. By implementing efficient, chunk-based data transfer protocols and a standardized connector interface, LMCache enables high-performance cross-query context sharing and cross-engine prefill-decode disaggregation, significantly reducing latency and improving throughput in enterprise settings.
Related Ideas
- Empirical Production Serving Benchmarking of MLA-DSA Attention with IndexShareTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🔴Hard
- Speculative KV-Cache Reuse for Interruptible InferenceAReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning🔴Hard
- Dynamic Bucket Resizing via Control TheoryPyTorch Distributed: Experiences on Accelerating Data Parallel Training🟡Fairly easy
- Non-Autoregressive Local Decoding for Real-Time InferenceHeartMuLa: A Family of Open Sourced Music Foundation Models🟠Medium
- Native Integration of Adaptive Gradient CompressionPyTorch Distributed: Experiences on Accelerating Data Parallel Training🟠Medium