Direct GPU Graph Recording and Persistent Memory Pointers to Eliminate Decode Dispatch Overhead
Adopt static computation graph capture and persistent memory pointer swapping to eliminate CPU-side computational graph construction overhead during decoding.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 6 and Appendix B, the authors identify that decoding speed is CPU-bound rather than storage-bound, with 44 ms per step consumed exclusively by CPU-side computational graph construction across the 40 decoder layers.
“The streaming engine’s decode is CPU-side, not storage-side: 44 ms per step of graph building in the 40-layer forward is the floor, and no storage-side optimization moves it.”
Limitation
Rebuilding and dispatching dynamic computational graphs at every single decode step imposes a hard latency floor of 44 ms that prevents the system from exceeding ~20 tok/s even when SSD prefetching is optimal.
Proposed Solution
Adopt static computation graph capture (such as CUDA Graphs or MLX-level static graph compilation with persistent slot references). Pre-record the entire forward pass graph once, keeping kernel launches and memory offsets invariant while updating only the underlying backing pointers of the sticky-slot expert memory buffers via pointer-swapping or zero-copy virtual address re-mapping.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Removing the 44 ms CPU graph construction tax per step could dramatically increase token throughput, potentially doubling generation speed on consumer Apple Silicon or PC hardware.
How to Get Started
How to test it: Hypothesis: Capturing a static computation graph with persistent sticky-slot pointer swapping eliminates the 44 ms per-step CPU graph-build floor and raises 35B decode speed above 25 tok/s. Baseline: Edge0-35B staged incr_stack on Mac mini M4 Pro 24 GB (20.4 tok/s); Metric: Median decode tok/s, per-step CPU dispatch latency, and element-wise relative L2 < 1% math contract parity; Stop if static graph capture breaks the <1% L2 parity contract or reduces total decode step latency by less than 15%.
- Profile the current decoding pipeline using PyTorch profiler or Apple Metal System Trace to isolate CPU-side graph construction bottlenecks.
- Implement CUDA Graphs capture or MLX static graph compilation for the model's forward pass loops.
- Develop a zero-copy virtual address re-mapping mechanism to update sticky-slot expert memory buffers without rebuilding the graph.
- Benchmark end-to-end token generation latency on consumer hardware or Apple Silicon M-series devices against the baseline FIFO implementation.
Code linked to this paper on Hugging Face: github.com/Edge0-AI/edge0
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The retrieved papers cover graph processing accelerators, memory safety, and GPU task abstractions, but none of them address static computation graph capture or pointer-swapping techniques for eliminating decode dispatch overhead in MoE or LLM serving frameworks.
We searched OpenAlex and arXiv for “Adopt static computation graph capture and persistent memory pointer swapping to eliminate CPU-side computational graph construction overhead during decoding.” and “Direct GPU Graph Recording and Persistent Memory Pointers to Eliminate Decode Dispatch Overhead” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing PredictionYu Lin, Yiming Wang, Runyuan Cai +2 more · 2026
The paper introduces EDGE0, a streaming inference framework designed to serve large mixture-of-experts (MoE) models (such as 35B and 8B variants) on consumer devices constrained by limited RAM. It overcomes SSD loading latency by using a cross-token prerouter that predicts routing one token ahead, executes the prediction directly as the actual routing to completely overlap SSD expert streaming with compute, and restores accuracy loss from int4 quantization and routing replacement via an unmerged distillation LoRA.
Related Ideas
- Hardware-Aware Kernels for Zero-Overhead InferenceBitNet b1.58 2B4T Technical Reportâ›”Very hard
- Lookup-Oriented Hardware Acceleration for Written Sparse Associative TablesWeight-Space Programming Doesn't Need Transformersđźź Medium
- Roofline-Anchored Hardware Counter Feedback to Break the Kernel Optimization CeilingAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUsđźź Medium
- CUDA/Triton Implementation of Element-wise LUTsBitnet.cpp: Efficient Edge Inference for Ternary LLMsđź”´Hard
- Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared AttentionMulti-Branch Transformerđźź Medium