Hybrid Kernel Scheduling for Prefill Optimization
Develop a dynamic execution engine with a hybrid scheduler that switches between parallel GEMM kernels for prefill and ELUT kernels for decoding.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The paper explicitly notes in the 'Limitations' section that Bitnet.cpp focuses on the decoding stage (memory-bound). However, the prefilling stage (processing the prompt) is computation-bound, where the resource bottleneck shifts.
Limitation
The current Element-wise LUT (ELUT) optimization is designed to minimize memory access, which is crucial for token generation (decoding). During the prefill phase, where matrix dimensions are larger and compute-bound, the overhead of LUT indexing and lack of dense compute optimizations makes the current kernels suboptimal compared to standard MAD (Multiply-Add) operations.
Proposed Solution
Develop a dynamic execution engine that detects the inference phase. Implement a hybrid scheduler that dispatches standard highly-parallelized MAD-based GEMM kernels for the computation-heavy prefill stage and switches to the proposed TL/ELUT kernels for the memory-bound decoding stage. This ensures optimal utilization of hardware for both prompt processing and token generation.
Why This Matters
Optimizing the prefill stage is critical for user experience (time-to-first-token) and overall throughput, especially for applications involving long prompts (RAG, document summarization).
How to Get Started
- Analyze the prefill and decoding execution phases of Bitnet.cpp on a standard GPU.
- Implement phase detection logic inside the inference runtime to monitor prompt processing stages.
- Integrate highly-parallelized MAD-based GEMM kernels for the computation-heavy prefill stage.
- Benchmark end-to-end inference latency against the decoding-only baseline using standard prompt lengths.
Code linked to this paper on Hugging Face: github.com/microsoft/BitNet
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed papers discuss prefill-decode scheduling, aggregation, and caching techniques such as SARATHI and POD-Attention, but none of them propose a hybrid execution engine switching between standard MAD-based GEMM kernels and TL/ELUT kernels for distinct inference phases.
- Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving · 2025 · 33 citations
- POD-A ttention : Unlocking Full Prefill-Decode Overlap for Faster LLM Inference · 2025 · 118 citations
- SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills · 2023 · 390 citations
We searched OpenAlex for “Develop a dynamic execution engine with a hybrid scheduler that switches between parallel GEMM kernels for prefill and ELUT kernels for decoding.” and “Hybrid Kernel Scheduling for Prefill Optimization” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
Bitnet.cpp: Efficient Edge Inference for Ternary LLMsJinheng Wang, Hansong Zhou, Ting Song +7 more · 2025
This paper introduces Bitnet.cpp, an inference library optimized for ternary (1.58-bit) Large Language Models (LLMs) like BitNet b1.58 on edge CPUs. It proposes a novel mixed-precision matrix multiplication (mpGEMM) library featuring Element-wise Lookup Table (TL) and "Int2 with a Scale" (I2_S) kernels to resolve spatial inefficiencies in existing bit-wise methods and achieve fast, lossless inference.
Related Ideas
- Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel OptimizationAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs🟠Medium
- Dynamic Block Sizing for Multi-Architecture ServingEfficient Memory Management for Large Language Model Serving with PagedAttention🔴Hard
- Predictive Semantic-Aware Preemption PolicyEfficient Memory Management for Large Language Model Serving with PagedAttention🟠Medium
- Joint Inter-DP and Intra-Sequence Layout Optimization for Multi-Document Sparse AttentionTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🟠Medium
- Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel OptimizationAscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs🟡Fairly easy