Skip to content

Source paper

MMFlash: Faster Speculative Decoding for Vision-Language Models

Sicheng Feng, Zigeng Chen, Song Wang and 2 more

Sicheng Feng, Zigeng Chen, Song Wang, Gongfan Fang, Xinchao Wang

arXiv 2611.18482 · 202639 upvotes on Hugging Face

Research ideas from this paper (5)

Generated by AI. The paper’s authors did not write these.

Query-Adaptive Video Context Compression and Frame-Level Visual Dependence for Long-Video Speculative Drafting

Select video draft tokens dynamically by question relevance and score visual dependence per frame so long-video drafters focus on informative moments instead of uniform strides.

MMFlash: Faster Speculative Decoding for Vision-Language Models
🟠Medium💡Moderate InnovationMultiple GPUsNo close match found#Multimodal#Efficiency

Concurrency- and Confidence-Adaptive Block-Size Scheduling for High-Batch VLM Serving

Adapt draft block lengths dynamically using serving batch size and predicted token acceptance confidence so speculative decoding avoids wasted verification compute under high concurrency.

MMFlash: Faster Speculative Decoding for Vision-Language Models
🟡Fairly easy🔧Minor TweakSingle GPU or lessNo close match found#Efficiency#Multimodal

Intra-Block Causal Conditioning and Uncertainty-Triggered Visual Refetching for Visually Grounded Spans

Add lightweight intra-block causal conditioning and uncertainty-triggered visual cross-attention to parallel diffusion drafters so tokens within visually grounded spans condition on earlier predictions.

MMFlash: Faster Speculative Decoding for Vision-Language Models
🔴Hard🔬Significant ResearchMultiple GPUsNo close match found#Multimodal#Efficiency

Single-Pass Visual Dependence Estimation to Eliminate Dual-Pass Target Scoring

Estimate token-level visual dependence from a single target forward pass via hidden-state shifts or attention statistics to eliminate the extra no-image scoring pass.

MMFlash: Faster Speculative Decoding for Vision-Language Models
🟡Fairly easy💡Moderate InnovationMultiple GPUsNo close match found#Training#Efficiency

Modality-Dependence Reweighting and Interleaved Sparse Attention for Omni-Modal Audio-Visual-Speech Models

Generalize modality-dependence loss reweighting and interleaved sparse attention to audio-visual and streaming speech models so speculative decoding accelerates omni-modal generation.

MMFlash: Faster Speculative Decoding for Vision-Language Models
🔴Hard🔬Significant ResearchMultiple GPUsNo close match found#Multimodal#Speech & audio

AI summary

MMFlash introduces a diffusion-based speculative decoding framework designed to accelerate inference in vision-language models (VLMs) while preserving exact target output distributions. After diagnosing that draft token acceptance drops sharply on visually dependent tokens even with multimodal training, the paper proposes a visually grounded loss reweighting scheme and a sparse attention pattern for long video contexts. Across seven image and three video benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58x on Qwen3.5 (and up to 4.71x on Qwen3.8-27B), outperforming autoregressive and text-focused diffusion drafters.

How it works (AI summary)

The framework employs a 5-layer lightweight block-diffusion drafter (block size 16) that omits raw visual tokens and instead reads tapped hidden states from 5 layers of the target VLM to propose 15 masked tokens in a single forward pass. During training, each token's cross-entropy loss is reweighted by the product of a verification-aware chain survival weight, a corpus-quantile visual dependence score derived from the KL divergence between target next-token distributions with and without visual input (scaled by target confidence), and the drafter's current prediction error probability. For long video inputs (~41K tokens), the drafter uses a static sparse attention pattern retaining 4 sink tokens, strided visual tokens (stride 32), all text tokens, and a 256-token local window, followed by 3-epoch fine-tuning on 2K video samples.