Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchNo close match found

Modality-Dependence Reweighting and Interleaved Sparse Attention for Omni-Modal Audio-Visual-Speech Models

Generalize modality-dependence loss reweighting and interleaved sparse attention to audio-visual and streaming speech models so speculative decoding accelerates omni-modal generation.

Time

a couple of months

Compute level

Multiple GPUs

Resources

Multi-GPU cluster or 8x H100/H200 node, open-source omni-modal target checkpoints (e.g., Qwen2.5-omni / Qwen3.5-omni), and public audio-visual-speech instruction datasets.

What the paper used

Training used 8 NVIDIA H100 GPUs on ~900K regenerated LLaVA-OV-1.5-Instruct-Data samples (6 epochs, batch size 2) plus 2K LLaVA-Video-178K samples (3 epochs); evaluation used a single NVIDIA H200 GPU across Qwen3.5-4B/9B, Gemma4-E4B-it, and Qwen3.8-27B.

Prerequisites

Omni-modal transformers, audio/speech tokenization, speculative decoding, and sparse attention.

Scope

New modality-dependence scoring, multi-stream drafter head design, and training/evaluation on omni-modal benchmarks.

The Gap

In Appendix F (Limitations & Future Work), the authors note that MMFlash is designed and evaluated solely for VLMs taking image or video inputs and generating text. Its visual dependence score (Equation 1) only contrasts with-image vs. without-image distributions, and its sparse attention pattern (Section 4.2) assumes contiguous visual tokens followed by text tokens.

“Another direction is to extend MMFlash to omni-modal models [59, 68], which also take audio as input and generate streaming speech.”
From the paper, Appendix F, page 15. Matched against the PDF text.

Limitation

Omni-modal architectures (such as Qwen2.5-omni and Qwen3.5-omni) take continuous audio alongside video in long interleaved contexts and generate streaming speech tokens in addition to text; text-only block-diffusion drafters and image-only KL reweighting cannot isolate audio-dependent or cross-modal audio-visual tokens nor handle interleaved multi-rate audio-visual KV caches.

Proposed Solution

Extend Equation 1 to multi-perturbation modality dependence by measuring the maximum or Shapley-style KL divergence of target next-token distributions under audio-ablated, visual-ablated, and jointly ablated prefixes, enabling separate reweighting of audio-grounded and joint audio-visual tokens. Pair this with an interleaved sparse attention mask that preserves temporal alignment across audio and video frames via synchronized striding and sink tokens, and adapt the block-diffusion drafter to output dual-stream text and discrete speech-codec tokens.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Extending modality-aware diffusion drafting to omni-modal models is expected to unlock lossless speculative decoding speedups for real-time voice and video assistants, where streaming speech latency is critical.

How to Get Started

How to test it: Test whether multi-perturbation modality-dependence reweighting improves draft acceptance length and end-to-end streaming speedup over unweighted DFlash and autoregressive baselines on an open-source omni-modal model (e.g., Qwen2.5-Omni-7B) across audio-visual QA and speech-generation benchmarks. Stop if speech-token acceptance remains too low (<2 tokens/block) to yield >1.3x real-time speedup.

  1. Profile token acceptance rates of an unweighted DFlash-style drafter on Qwen2.5-Omni-7B across audio-only, video-only, and joint audio-visual QA tasks, grouped by audio- and visual-ablation KL quartiles.
  2. Extend MMFlash's loss reweighting to use audio-ablated and visual-ablated KL dependence scores on regenerated audio-visual training samples.
  3. Design an interleaved audio-visual sparse attention pattern that retains synchronized audio-video anchors, all text tokens, and a local decoding window.
  4. Evaluate lossless text and speech-codec decoding speedups (tok/s and time-to-first-audio-chunk) under single-request and concurrent serving.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

General omni-modal models and benchmarks like OmniJigsaw, LongCat-Flash-Omni, and Hallo-Live are surveyed or introduced in recent literature, but none evaluate Shapley-style modality-dependence reweighting or interleaved sparse attention for dual-stream text and speech-codec block-diffusion drafters.

We searched OpenAlex and arXiv for “speculative decoding omni modal audio visual streaming speech” and “Modality-Dependence Reweighting and Interleaved Sparse Attention for Omni-Modal Audio-Visual-Speech Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

MMFlash: Faster Speculative Decoding for Vision-Language Models

Sicheng Feng, Zigeng Chen, Song Wang +2 more · 2026

MMFlash introduces a diffusion-based speculative decoding framework designed to accelerate inference in vision-language models (VLMs) while preserving exact target output distributions. After diagnosing that draft token acceptance drops sharply on visually dependent tokens even with multimodal training, the paper proposes a visually grounded loss reweighting scheme and a sparse attention pattern for long video contexts. Across seven image and three video benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58x on Qwen3.5 (and up to 4.71x on Qwen3.8-27B), outperforming autoregressive and text-focused diffusion drafters.

Related Ideas

Is this a good idea?

Share onXLinkedIn