Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared Attention
Combine branch-wise KV sharing with cross-layer KV caching to eliminate redundant key-value memory traffic across both network depth and parallel architectural branches.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 2.3 and Section 6 discuss KV-cache sharing and future directions. Joint-KV shares KV states purely across parallel branches at the same depth level (intra-layer, across-branch), while keeping KV caches separate across depth. However, long-context decoding memory consumption in the shared prelude and coda remains uncompressed, and branch layers still require separate KV projections at each sequential step.
Limitation
Joint-KV restricts cache reduction strictly to the branch dimension (1/K cache reduction within the branched stack only), leaving sequential depth layers uncompressed and yielding overall model KV savings capped at 19% to 34%.
Proposed Solution
Formulate a dual-axis KV sharing framework combining Joint-KV with cross-layer attention (CLA) and decoder-decoder architectures (YOCO). Parameterize key-value projections such that shared branch representations at depth l are reused across subsequent depths (l + 1 to l + m) or retained as a single persistent cache throughout the branch region. Develop custom FlashAttention kernels that accommodate folded branch queries attending over a single cross-depth, cross-branch shared KV tensor.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Could boost KV cache memory reduction from ~30% to over 70% during long-context serving, significantly increasing maximum batch concurrency without degrading language modeling perplexity.
How to Get Started
How to test it: Pretrain a 0.8B (W1 scale) Balanced MBT with 2-layer CLA sharing across branch depths on 28B tokens of FineWeb-Edu. Compare validation loss and H200 decoding throughput against the baseline Joint-KV Balanced MBT. Terminate the project if validation perplexity degrades by more than 0.05 or decoding latency increases due to cross-layer synchronization overhead.
- Clone the released Multi-Branch Transformer codebase and verify reproducing the baseline W1 Balanced MBT configuration.
- Implement a layer-linking module that routes the shared Joint-KV cache of branch depth l to depth l+1.
- Update the nano-vLLM head folding engine to read cached KV blocks across paired layers.
- Train small proxy models on 5B tokens to verify gradient stability and initial validation loss.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers and DepthWeave-KV explore cross-layer KV cache compression and fusion techniques, while SparseX investigates segment-level KV sharing. However, none of these listed results combine Joint-KV, cross-layer attention, and custom FlashAttention kernels for folded branch queries into a unified dual-axis sharing framework.
- Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers · 2025 · 7 citations
- DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression · 2026 · 1 citations
- SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving · 2026 · 2 citations
We searched OpenAlex and arXiv for “cross layer attention joint kv cache transformer” and “Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared Attention” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Multi-Branch TransformerGaotang Li, Lingjie Chen, Chunyuan Deng +6 more · 2026
This paper introduces the Multi-Branch Transformer (MBT), an architectural paradigm that parallelizes the sequential execution of Transformer blocks by routing representations from shared prelude layers through multiple independently parameterized parallel branches before fusing them in shared coda layers. To optimize inference at long contexts, the authors introduce Joint-KV, which computes a unified key-value cache across concurrent branches per depth level, alongside branch grouping and query head folding within an optimized inference runtime. Experiments on 0.8B to 2.1B parameter MoE models show that MBTs consistently improve language modeling loss and zero-shot benchmark performance under matched compute, while reducing KV cache storage up to 34% and increasing decode throughput on an H200 GPU by up to 51.4%.
Related Ideas
- Global Content-Addressable KV DeduplicationEfficient Memory Management for Large Language Model Serving with PagedAttention🟠Medium
- Speculative KV-Cache Reuse for Interruptible InferenceAReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning🔴Hard
- Prompt-Relevant Historical Memory Retrieval and Entropy-Calibrated Layer-Head KV CompressionMovieRoll: A Systematic Framework for Rolling Out Long-Form Videos🟡Fairly easy
- Efficient Input-Dependent Queries for Attention ResidualsAttention Residuals🟠Medium
- Granular Importance-Aware Sparse RetrievalLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference🔴Hard