State-Space Memory Integration for Unbounded Audio Context
Replace static audio sink tokens with a recurrent state-space model attached to the audio stream prefix to manage unbounded audio context without Softmax collapse.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
During Stage II (Causal ODE Regression), the extreme audio token sparsity causes conditional distribution shifts and Softmax collapse. The paper fixes this by prepending 16 learnable Audio Sink Tokens with Identity RoPE to artificially expand the attention denominator.
“This operation successfully breaks the extremely sharp Softmax collapse and restores attention entropy, thereby acting as a contextual buffer to absorb anomalous logit perturbations and quelling the gradient storm at its source.”
Limitation
Audio Sink Tokens act as a static architectural patch that absorbs gradient variance but does not truly capture or compress long-term historical audio context, limiting the model's ability to maintain complex acoustic coherence over very long generation rollouts.
Proposed Solution
Replace the static Audio Sink Tokens with a hybrid State-Space Model (SSM) or Mamba-like recurrence attached to the audio stream's global prefix. Distill the bidirectional long-term audio context into this recurrent state vector, allowing it to dynamically update and pass historical acoustic features to the causal attention blocks without memory explosion.
Why This Matters
This would drastically improve long-term acoustic consistency (e.g., maintaining the same speaker identity or background ambiance over minute-long generations) without needing an infinitely scaling KV-cache.
How to Get Started
How to test it: Test whether replacing static Audio Sink Tokens with an SSM/Mamba recurrent state on the audio stream stabilizes Stage II training and improves long-rollout acoustic coherence over the S=16 Sink Token Identity RoPE baseline. Track Stage II max gradient norm and denoising loss alongside long-horizon (30-60s) FAD, CLAP, TA-IB, DeSync, and FPS, stopping if Stage II suffers Softmax collapse/NaNs, fails to beat S=16 on long-rollout FAD/CLAP, or drops inference below 24 FPS.
- Integrate a Mamba or state-space model recurrence layer into the global prefix of the audio generation stream.
- Remove the static audio sink tokens and their identity RoPE configurations from the attention denominator.
- Set up a distillation pipeline to transfer bidirectional long-term audio context into the recurrent state vector.
- Measure token sparsity stability and generation quality on a large compute cluster.
Code linked to this paper on Hugging Face: github.com/OmniForcing/OmniForcing
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed references explore state-space models for audio generation and memory patterns in state space models, such as It's Raw! Audio Generation with State-Space Models and MemMamba: Rethinking Memory Patterns in State Space Model. They do not examine the specific integration of replacing static Audio Sink Tokens with a hybrid SSM attached to an audio stream's global prefix for causal attention blocks.
- It's Raw! Audio Generation with State-Space Models · 2022 · 241 citations
- MemMamba: Rethinking Memory Patterns in State Space Model · 2025 · 6 citations
We searched OpenAlex for “Replace static audio sink tokens with a recurrent state-space model attached to the audio stream prefix to manage unbounded audio context without Softmax collapse.” and “State-Space Memory Integration for Unbounded Audio Context” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
OmniForcing: Unleashing Real-time Joint Audio-Visual GenerationYaofeng Su, Yuming Li, Zeyue Xue +7 more · 2026
This paper introduces OmniForcing, the first framework to distill a heavy, bidirectional dual-stream audio-visual diffusion model into a real-time streaming autoregressive generator. By mitigating training instabilities caused by severe modality asymmetry and token sparsity, the framework achieves state-of-the-art streaming generation at 25 FPS on a single GPU while preserving synchronization and visual quality.
Related Ideas
- Homogeneous Mamba-Based Audio TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🔴Hard
- Dynamic Attention-Based Identity RoutingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟡Fairly easy
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Resolving Speaker Drift in Long-Form Synthesis via Explicit Global ConditioningTADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment🟠Medium
- Infinite-Context Dual-Autoregressive Architecture for Unbounded Audiobook SynthesisFish Audio S2 Technical Report🔴Hard