Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchNo close match found

Causal-SLA for Autoregressive Video Generation

Develop a causal Sparse-Linear Attention kernel and apply speculative decoding to enable autoregressive video generation.

Time

a couple of months

Compute level

Large cluster

Resources

Large compute cluster

Prerequisites

transformer architectures

Scope

new model architecture

The Gap

The Conclusion (Section 3) notes the framework is currently limited to diffusion and plans to extend to 'autoregressive video diffusion.' The current Sparse-Linear Attention (SLA) is designed for bidirectional contexts in diffusion transformers.

Limitation

The current attention optimizations rely on sparsity patterns that do not account for the strict causal masking required in autoregressive (next-token prediction) video models.

Proposed Solution

Develop a 'Causal-SLA' kernel that enforces causal masking within the sparse attention computation. Replace the rCM diffusion distillation with Speculative Decoding or similar token-level acceleration techniques suitable for sequential generation.

Why This Matters

Extends the massive speedups of TurboDiffusion to the other major class of video generation models (like Sora-style AR models), broadening the framework's impact.

How to Get Started

  1. Design the causal masking logic for the Sparse-Linear Attention computation.
  2. Implement the custom attention kernel in CUDA or Triton for efficient execution.
  3. Integrate speculative decoding techniques to accelerate sequential token or frame generation.
  4. Train and validate the autoregressive video generation framework on a large cluster.

Code linked to this paper on Hugging Face: github.com/thu-ml/TurboDiffusion

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

The listed references discuss speculative decoding and sparse-linear attention mechanisms such as SLA2 and MInference 1.0, but none combine them into a 'Causal-SLA' kernel specifically for autoregressive video generation as proposed.

We searched OpenAlex for “Develop a causal Sparse-Linear Attention kernel and apply speculative decoding to enable autoregressive video generation.” and “Causal-SLA for Autoregressive Video Generation” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times

Jintao Zhang, Kaiwen Zheng, Kai Jiang +5 more · 2025

The paper introduces TurboDiffusion, a framework designed to accelerate Wan video diffusion models by 100-200x on NVIDIA RTX 5090 GPUs. It achieves this by combining four optimizations: low-bit attention (SageAttention), sparse-linear attention (SLA), step distillation via regularized Consistency Models (rCM), and W8A8 quantization of linear layers.

Related Ideas

Is this a good idea?

Share onXLinkedIn