Dynamic Semantic Attention Sinks for Long-Horizon Streaming
Replace fixed attention sinks with a learnable token eviction policy that dynamically evaluates the semantic importance of past key-value pairs.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 4 introduces 'Anchor-Heavy Identity Sinks' (AHIS) to preserve identity by hard-coding the first few blocks of the video as permanent attention sinks.
Limitation
Hard-coding the initial blocks as sinks assumes the starting state is the perfect reference for the entire session. This fails to adapt to dynamic changes (e.g., lighting shifts, changing accessories) and cannot recover if the initial frames are suboptimal.
Proposed Solution
Replace fixed sink indices with a 'Learnable Token Eviction' policy. Implement a lightweight scoring module within the attention mechanism that dynamically evaluates the 'semantic importance' of past KV pairs, allowing the model to update its long-term memory bank with high-quality representations from later in the stream.
Why This Matters
This would allow for truly indefinite streaming where the avatar can evolve or adapt to new contexts without being tethered to potentially stale or low-quality initial frames.
How to Get Started
- Review long-horizon video generation and streaming attention codebase architectures.
- Design a lightweight scoring module to calculate semantic importance scores for historical KV pairs.
- Evaluate the learnable eviction policy against fixed anchor-heavy identity sinks on long-video generation tasks.
Code linked to this paper on Hugging Face: github.com/GAIR-NLP/LiveTalk
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as 'Learning to Evict from Key-Value Cache' and 'DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation' explore learnable token eviction policies and dynamic semantic sinks to manage long-context and streaming KV caches. These works cover the core proposal of replacing fixed sink indices with dynamic scoring mechanisms, leaving little distinction for the proposed method.
- Learning to Evict from Key-Value Cache · 2026 · 13 citations
- Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction · 2026 · 13 citations
- Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs · 2025 · 20 citations
- DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation · 2026 · 7 citations
We searched OpenAlex for “Replace fixed attention sinks with a learnable token eviction policy that dynamically evaluates the semantic importance of past key-value pairs.” and “Dynamic Semantic Attention Sinks for Long-Horizon Streaming” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy DistillationEthan Chern, Zhulin Hu, Bohao Tang +4 more · 2025
The paper presents LiveTalk, a real-time system for interactive multimodal avatar video generation using a novel video diffusion distillation recipe. By refining multimodal data quality, ensuring full ODE convergence before distillation, and using an aggressive optimization schedule, the authors achieve 20x acceleration and sub-second latency while maintaining visual quality comparable to larger bidirectional models.
Related Ideas
- Dynamic Attention-Based Identity RoutingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟡Fairly easy
- Hierarchical Dynamic Memory Aggregation for Long-Form Video ReasoningV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟠Medium
- Persistent Identity Memory ModuleStable Video Infinity: Infinite-Length Video Generation with Error Recycling🟡Fairly easy
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Infinite-Horizon Attention via Compressed Memory Caching in Hierarchical DenoisingHiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising🔴Hard