Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel Optimization
Extend kernel optimization evaluation to dynamic tensor shapes, fused multistage operators, and end-to-end model throughput so agents cannot overfit single static input configurations.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.1 and Appendix E evaluate only the 17 basic memory-access kernels in Batch 1, leaving the 31 multistage and fused attention kernels in Batches 2 and 3 unreported. Moreover, each task test case binds a kernel to a single static deployment configuration (for example, AddRMSNorm fixed to batch size 1 and sequence length 32 in Qwen3-0.6B prefill) and measures kernel latency in isolation from surrounding graph execution.
“This report evaluates the 17 kernel optimization tasks in Batch 1; results for Batches 2 (17 kernels) and 3 (14 kernels) will be reported in future updates.”
Limitation
Evaluating isolated basic kernels on a single static tensor shape allows agents to over-specialize code for one input configuration while failing to reflect dynamic prefill/decode shape regimes, complex fused operators, or end-to-end model throughput.
Proposed Solution
Complete the evaluation benchmark across Batches 2 and 3 and augment each task manifest with a distribution of real workload shapes (varying batch sizes, prefill and decode sequence lengths, and tensor-parallel configurations) weighted by profiling invocation frequency. Add an end-to-end replay harness inside AscendForge using vLLM-Ascend and MindSpeed-LLM execution traces to verify that kernel-level speedups translate to full-model step-time reductions without regressing non-target shapes.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
If successful, this extension would prevent shape overfitting and ensure that agent-optimized kernels deliver reliable speedups across realistic production inference and training workloads.
How to Get Started
How to test it: Hypothesis: single-shape Batch 1 kernel optimizations degrade or fail to generalize on unseen prefill/decode shapes and fused Batch 2/3 operators, whereas multi-shape-evaluated candidates sustain positive geometric mean speedup. Baseline: single-shape Batch 1 winners from Claude Code/Opus 5 and Codex/GPT-5.6; metric: frequency-weighted multi-shape GM speedup and end-to-end vLLM-Ascend step latency; stop if multi-shape evaluation adds over 5x NPU profiling overhead without altering kernel rankings (Spearman rho > 0.95).
- Clone the released AscendGYM repository and verify baseline compilation and execution on Ascend 910C for all 48 tasks across Batches 1, 2, and 3.
- Profile Qwen3-0.6B and DeepSeek-V4 inference and training runs in vLLM-Ascend and MindSpeed-LLM to extract dynamic prefill and decode tensor shape distributions for each operator.
- Extend the AscendForge test driver and input manifests to score candidates using a workload-frequency-weighted geometric mean across multiple tensor shapes.
- Re-evaluate the top Batch 1 kernels optimized by Claude Code/Opus 5 and Codex/GPT-5.6 against the multi-shape suite and end-to-end forward passes to measure shape overfitting.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like BladeDISC and TVM cover end-to-end compiler optimizations and workload evaluations, while work on Ascend NPUs touches upon dynamic shapes and operator optimization. However, none of the listed results combine a full evaluation benchmark across multiple batches with workload-frequency weighting and vLLM-Ascend/MindSpeed-LLM replay harnesses for NPU kernels.
- XY-Serve: End-to-End Versatile Production Serving for Dynamic LLM Workloads · 2025 · 11 citations
- BladeDISC: Optimizing Dynamic Shape Machine Learning Workloads via Compiler Approach · 2023 · 32 citations
- Adaptive Matrix Multiplication for Dynamic Shapes on Ascend NPUs · 2026 · 0 citations
- AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization · 2026 · 7 citations
We searched OpenAlex and arXiv for “kernel optimization benchmark dynamic shapes end-to-end workload evaluation” and “Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel Optimization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUsAI Training Platform Team, Bingwei Zhang, Jing Li +24 more · 2026
This paper introduces AscendGYM, a benchmark and execution sandbox (AscendForge) for evaluating how effectively LLM coding agents optimize existing Ascend NPU kernel implementations from real-world LLM training and inference workloads. Unlike prior kernel-generation benchmarks, AscendGYM measures optimization gains relative to production baselines alongside token cost using a new Speedup-Token Score (STS) metric.
Related Ideas
- Feature-Parity Verification and Latency-Accuracy Co-Evaluation for Served Frontier VLAsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🟡Fairly easy
- Hybrid Kernel Scheduling for Prefill OptimizationBitnet.cpp: Efficient Edge Inference for Ternary LLMs🟡Fairly easy
- Hardware-Efficient Backbone for Scalable DeploymentUtonia: Toward One Encoder for All Point Clouds⛔Very hard
- Joint Inter-DP and Intra-Sequence Layout Optimization for Multi-Document Sparse AttentionTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🟠Medium
- Token-Normalized and Convergence-Controlled Evaluation of VLM Scaling Laws in RoboticsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🟡Fairly easy