Prompt-Relevant Historical Memory Retrieval and Entropy-Calibrated Layer-Head KV Compression
Replace fixed-rule cache eviction with prompt- and visual-relevance retrieval and entropy-calibrated per-head compression to reduce semantic drift over minute-long video rollouts.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Sections 3.3, 3.6, and Appendix A.5, HieraKV applies a uniform binary rule across all attention heads—retaining full history in blocks 1-6 and only sink plus 1 recent chunk in blocks 7-54—while threshold-triggered KV recomputation fires every 18 chunks (or on prompt switch), retaining only the sink, middle chunks from the last 5 events, and 3 recent chunks, re-encoded at t=0 without text conditioning.
“Despite high full-video naturalness, the distilled model receives lower semantic and drifting-naturalness ratings at the longest evaluated duration than at the shortest.”
Limitation
Fixed recency heuristics permanently discard earlier-event entities beyond the 5-event window, omit text context during cache rebuilds, and ignore head-level functional differences within blocks, contributing to semantic alignment and drifting-naturalness degradation at 60s.
Proposed Solution
Replace the static 5-event middle selection during KV recomputation with a relevance-scored retriever that compares candidate historical chunk visual keys/pooled embeddings against the active event prompt (via cross-modal text-visual similarity) and recent query states, pass active prompt embeddings during causal KV re-encoding, and calibrate per-head (rather than per-block) full-history vs. sink-recent budgets using offline attention-entropy profiling.
Why This Matters
Selecting prompt-relevant history and finer per-head cache budgets should preserve recurring characters or objects across long narrative arcs and reduce semantic drift without increasing peak KV memory.
How to Get Started
How to test it: Test whether prompt-relevance chunk selection, text-conditioned recomputation, and per-head entropy-calibrated HieraKV improve 60s single-event and 6-event (plus 12-event entity-reentry) HeliosBench Semantic Alignment and Drifting Naturalness at equal or lower KV memory vs. MovieRoll-Distilled's fixed policy; stop if retrieval overhead adds >5% latency without raising Semantic or Drifting Naturalness ratings.
- Profile attention entropy across all 54 blocks x 16 heads on 32 MovieRoll-Distilled 60s rollouts to identify head-level global vs. local specialization.
- Implement an equal-memory per-head HieraKV allocation mask and compare against the default 6-global/48-local block split on 30s and 60s HeliosBench.
- Augment the KV recomputation selector with cosine-similarity scoring between stored chunk representations and the current prompt/query embeddings, enabling text-conditioned re-encoding.
- Evaluate on 60s MemFlow prompts plus a constructed 60s entity-reintroduction test set where an early-event subject returns after 5 intervening events.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Streaming video question-answering papers like StreamKV and OmniMem investigate KV cache retrieval and compression for video, yet the proposal to combine relevance-scored cross-modal text-visual key retrieval with offline attention-entropy profiled layer-head budgeting is distinct.
- StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression · 2025 · 21 citations
- InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding · 2025 · 71 citations
- OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation · 2026 · 10 citations
We searched OpenAlex and arXiv for “adaptive KV cache compression retrieval long video generation” and “Prompt-Relevant Historical Memory Retrieval and Entropy-Calibrated Layer-Head KV Compression” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
MovieRoll: A Systematic Framework for Rolling Out Long-Form VideosYunyang Ge, Weijie Kong, Ryan Xu +9 more · 2026
MovieRoll introduces a base-to-distilled framework for streaming long-form video generation that sustains visual quality, active motion, and prompt continuity up to 60 seconds while bounding historical KV-cache memory. By combining parallel rolling-window fine-tuning, hierarchical layer-wise cache compression, cumulative long-rollout distribution matching distillation, and streaming decoding infrastructure, it outperforms existing streaming baselines on single-event and multi-event HeliosBench evaluations.
Related Ideas
- Adaptive Attention-Based Memory CompressionHelios: Real Real-Time Long Video Generation Model🟠Medium
- Importance-Weighted Semantic Memory Compaction for Ultra-Long Horizon TasksGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟡Fairly easy
- Efficient Input-Dependent Queries for Attention ResidualsAttention Residuals🟠Medium
- Semantic Compressive Memory for Long-Horizon ContextMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium
- Query-Adaptive Video Context Compression and Frame-Level Visual Dependence for Long-Video Speculative DraftingMMFlash: Faster Speculative Decoding for Vision-Language Models🟠Medium