Skip to content

Source paper

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs

AI Training Platform Team, Bingwei Zhang, Jing Li and 24 more

AI Training Platform Team, Bingwei Zhang, Jing Li, Amir Abdullin, Sihang Chen, Xin Chen, Liangjun Feng, Yingmeng Gao, Tao Guo, Junjie Huang, Dongfang Li, Zhanglian Lin, Jiajun Liu, Zekun Liu, Zhengxuan Lu, Xiaodong Luo, Jian Meng, Ivan Nosov, Linyuan Qiu, Weijian Sun, Wanrui Yan, Binfan Zheng, Zhicheng Zhou, Baotian Hu, Min Zhang, Haizhou Li, Zhiquan Luo

arXiv 2611.17621 · 202692 upvotes on Hugging Face

Research ideas from this paper (5)

Generated by AI. The paper’s authors did not write these.

Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel Optimization

Extend kernel optimization evaluation to dynamic tensor shapes, fused multistage operators, and end-to-end model throughput so agents cannot overfit single static input configurations.

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs
🟠Medium💡Moderate InnovationMultiple GPUsRelated work found#Efficiency#Data & evaluation

Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel Optimization

Incorporate compilation, profiling, and hardware execution wall-clock time into the optimization score so agents are rewarded for fast turnaround rather than low token usage alone.

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs
🟡Fairly easy💡Moderate InnovationSingle GPU or lessNo close match found#Data & evaluation#Efficiency

Roofline-Anchored Hardware Counter Feedback to Break the Kernel Optimization Ceiling

Feed automated hardware roofline headroom and memory-stall diagnostics into the optimization loop so coding agents identify architectural bottlenecks instead of guessing code edits.

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs
🟠Medium🔬Significant ResearchMultiple GPUsRelated work found#Agents#Efficiency

Retrieval-Augmented Cross-Kernel Experience Transfer to Mitigate Agent Harness Sensitivity

Replace fixed heuristic lesson imports with computation-matched experience retrieval so optimization insights transfer reliably across kernels and reduce sensitivity to agent scaffolding.

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs
🟡Fairly easy💡Moderate InnovationMultiple GPUsRelated work found#Agents#Retrieval & memory

Multi-Seed Stability Analysis and Distributed Multi-NPU Collective Kernel Benchmarking

Add multi-seed trajectory repeats and multi-device collective workloads to the evaluation protocol so benchmark rankings capture search variance and distributed communication bottlenecks.

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs
🔴Hard💡Moderate InnovationLarge clusterNo close match found#Data & evaluation#Efficiency

AI summary

This paper introduces AscendGYM, a benchmark and execution sandbox (AscendForge) for evaluating how effectively LLM coding agents optimize existing Ascend NPU kernel implementations from real-world LLM training and inference workloads. Unlike prior kernel-generation benchmarks, AscendGYM measures optimization gains relative to production baselines alongside token cost using a new Speedup-Token Score (STS) metric.

How it works (AI summary)

AscendGYM comprises 48 kernel optimization tasks across three difficulty batches (evaluating the 17 memory-access tasks in Batch 1 in this report), pairing baseline Ascend C code with real workload tensor shapes, golden CPU reference outputs, and an Ascend 910C execution sandbox (AscendForge) that automates compilation, correctness checks, and msprof profiling. It supports two modes—evaluating external coding harnesses (Claude Code, Codex CLI, OpenCode) paired with frontier LLMs, and evaluating models under a fixed AuraKernel-based Default harness—scored by STS, Cost-to-Gain (CTG), GM speedup, fast_p, token/round/wall-clock budgets, and pass rates.