Look-Ahead Causal Attention for Enhanced Cross-Modal Synchrony
Implement a bounded look-ahead attention mechanism with a localized temporal buffer to improve text-consistency and audio-video synchrony in causal generative models.
Source paper
Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan
Generated by AI. The paper’s authors did not write these.
Implement a bounded look-ahead attention mechanism with a localized temporal buffer to improve text-consistency and audio-video synchrony in causal generative models.
Develop a continuous-time causal masking formulation using time-aware rotary embeddings and fractional thresholding to align asymmetric video and audio modalities dynamically.
Replace static audio sink tokens with a recurrent state-space model attached to the audio stream prefix to manage unbounded audio context without Softmax collapse.
This paper introduces OmniForcing, the first framework to distill a heavy, bidirectional dual-stream audio-visual diffusion model into a real-time streaming autoregressive generator. By mitigating training instabilities caused by severe modality asymmetry and token sparsity, the framework achieves state-of-the-art streaming generation at 25 FPS on a single GPU while preserving synchronization and visual quality.
OmniForcing uses a three-stage distillation pipeline (Bidirectional DMD, Causal ODE Regression, and Joint Self-Forcing) to convert an offline diffusion model into a causal engine. Key innovations include Asymmetric Block-Causal Alignment with a Global Prefix to handle VAE stride mismatch, an Audio Sink Token mechanism with Identity RoPE to prevent gradient explosion from token sparsity, and a Modality-Independent Rolling KV-Cache for parallel inference.