Causal-SLA for Autoregressive Video Generation
Develop a causal Sparse-Linear Attention kernel and apply speculative decoding to enable autoregressive video generation.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The Conclusion (Section 3) notes the framework is currently limited to diffusion and plans to extend to 'autoregressive video diffusion.' The current Sparse-Linear Attention (SLA) is designed for bidirectional contexts in diffusion transformers.
Limitation
The current attention optimizations rely on sparsity patterns that do not account for the strict causal masking required in autoregressive (next-token prediction) video models.
Proposed Solution
Develop a 'Causal-SLA' kernel that enforces causal masking within the sparse attention computation. Replace the rCM diffusion distillation with Speculative Decoding or similar token-level acceleration techniques suitable for sequential generation.
Why This Matters
Extends the massive speedups of TurboDiffusion to the other major class of video generation models (like Sora-style AR models), broadening the framework's impact.
How to Get Started
- Design the causal masking logic for the Sparse-Linear Attention computation.
- Implement the custom attention kernel in CUDA or Triton for efficient execution.
- Integrate speculative decoding techniques to accelerate sequential token or frame generation.
- Train and validate the autoregressive video generation framework on a large cluster.
Code linked to this paper on Hugging Face: github.com/thu-ml/TurboDiffusion
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed references discuss speculative decoding and sparse-linear attention mechanisms such as SLA2 and MInference 1.0, but none combine them into a 'Causal-SLA' kernel specifically for autoregressive video generation as proposed.
- Speculative Decoding and Beyond: An In-Depth Survey of Techniques · 2025 · 32 citations
- SLA2: Sparse-Linear Attention with Learnable Routing and Quantization-aware Training · 2026 · 26 citations
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention · 2024 · 510 citations
We searched OpenAlex for “Develop a causal Sparse-Linear Attention kernel and apply speculative decoding to enable autoregressive video generation.” and “Causal-SLA for Autoregressive Video Generation” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
TurboDiffusion: Accelerating Video Diffusion Models by 100-200 TimesJintao Zhang, Kaiwen Zheng, Kai Jiang +5 more · 2025
The paper introduces TurboDiffusion, a framework designed to accelerate Wan video diffusion models by 100-200x on NVIDIA RTX 5090 GPUs. It achieves this by combining four optimizations: low-bit attention (SageAttention), sparse-linear attention (SLA), step distillation via regularized Consistency Models (rCM), and W8A8 quantization of linear layers.
Related Ideas
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Direct Causal-Mode Distribution Matching for Autoregressive Video DistillationHiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising🔴Hard
- Infinite-Horizon Attention via Compressed Memory Caching in Hierarchical DenoisingHiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising🔴Hard
- Motion-Aware Transition Modeling via Multi-Frame ConditioningStoryMem: Multi-shot Long Video Storytelling with Memory🟡Fairly easy
- Look-Ahead Causal Attention for Enhanced Cross-Modal SynchronyOmniForcing: Unleashing Real-time Joint Audio-Visual Generation🟠Medium