Concurrency- and Confidence-Adaptive Block-Size Scheduling for High-Batch VLM Serving
Adapt draft block lengths dynamically using serving batch size and predicted token acceptance confidence so speculative decoding avoids wasted verification compute under high concurrency.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 5.3 (Table 3) and Appendix E.1 (Table 8), MMFlash is served with a fixed block size of 16 across all concurrency levels (1 to 32) on SGLang and vLLM. In the test-time block-size ablation (Table 4), the paper tests fixed block sizes of 8 and 16 and notes that smaller blocks preserve speedups well (only a 5.3% drop at concurrency 1) while capping maximum acceptance.
“Although the speedup narrows as decoding becomes compute-bound, MMFlash at concurrency 16 still delivers higher throughput than the baseline at concurrency 32 on all six tasks, confirming its effectiveness in real-world serving.”
Limitation
As serving concurrency increases from 1 to 32, target verification shifts from memory-bandwidth-bound to compute-bound, causing MMFlash's speedup to drop sharply (e.g., from 3.46x to 1.61x on CharXiv for Qwen3.5-4B, and from 2.67x to 1.14x on MMStar for Qwen3.5-9B); verifying 15 speculative tokens per request at high batch sizes or during low-acceptance visually dependent spans wastes target FLOPs on rejected suffixes.
Proposed Solution
Introduce a runtime block-size scheduler inside the SGLang/vLLM speculative runner that maps current batch concurrency and the drafter's predicted token survival probabilities (or step-1/2 entropy) to an optimal block size B in {4, 8, 12, 16}. Because the block-diffusion drafter already generalizes across smaller test-time block sizes without retraining, the scheduler can cap draft mask positions when concurrency enters the compute-bound regime or when visual-grounding uncertainty is high, while using B=16 for low-concurrency or text-heavy spans.
Why This Matters
Dynamic block-size scheduling is expected to recover throughput and reduce verification FLOPs at medium-to-high serving concurrency (8-32+ requests) without requiring drafter retraining or sacrificing single-request latency.
How to Get Started
How to test it: Test whether an adaptive block-size policy (choosing B in {4, 8, 12, 16} from batch concurrency and drafter first-token confidence) beats fixed B=16 and fixed B=8 MMFlash in decode throughput (tok/s) across concurrencies 1 to 32 on CharXiv, MMStar/MMBench, and DynaMath. Stop if adaptive scheduling fails to improve throughput over fixed B=16 at concurrency >= 16 or degrades concurrency-1 throughput.
- Reproduce Table 3 on SGLang (and Table 8 on vLLM) with fixed block sizes B=4, 8, 12, and 16 across concurrencies 1, 4, 8, 16, and 32 to map the Pareto frontier of throughput vs. block size.
- Profile drafter forward pass and target verification latency separately across batch sizes to fit a lightweight compute-bound cost model.
- Implement a dynamic block-size controller in the SGLang/vLLM speculative decoding loop conditioned on active batch size and drafter early-position confidence.
- Evaluate end-to-end throughput and acceptance rate on mixed-task serving traces combining high-acceptance (CharXiv/DynaMath) and low-acceptance (MMStar/MMBench) requests.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as Nightjar and FASER discuss dynamic speculative decoding for LLM serving, while DFlash introduces block diffusion for speculative decoding, but none specifically combine batch concurrency with drafter survival probabilities to schedule block-diffusion sizes in VLM serving.
- DFlash: Block Diffusion for Flash Speculative Decoding · 2026 · 156 citations
- Nightjar: Dynamic adaptive speculative decoding for large language models serving · 2026 · 7 citations
- FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving · 2026 · 4 citations
- DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation · 2026 · 108 citations
We searched OpenAlex and arXiv for “dynamic block size speculative decoding high concurrency serving” and “Concurrency- and Confidence-Adaptive Block-Size Scheduling for High-Batch VLM Serving” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
MMFlash: Faster Speculative Decoding for Vision-Language ModelsSicheng Feng, Zigeng Chen, Song Wang +2 more · 2026
MMFlash introduces a diffusion-based speculative decoding framework designed to accelerate inference in vision-language models (VLMs) while preserving exact target output distributions. After diagnosing that draft token acceptance drops sharply on visually dependent tokens even with multimodal training, the paper proposes a visually grounded loss reweighting scheme and a sparse attention pattern for long video contexts. Across seven image and three video benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58x on Qwen3.5 (and up to 4.71x on Qwen3.8-27B), outperforming autoregressive and text-focused diffusion drafters.
Related Ideas
- Dynamic Block Sizing for Multi-Architecture ServingEfficient Memory Management for Large Language Model Serving with PagedAttention🔴Hard
- Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel OptimizationAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs🟠Medium
- Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel OptimizationAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs🟡Fairly easy
- Adaptive Block Sizing for Memory-Efficient Attention ResidualsAttention Residuals🔴Hard
- Hybrid Kernel Scheduling for Prefill OptimizationBitnet.cpp: Efficient Edge Inference for Ternary LLMs🟡Fairly easy