Hybrid Kernel Scheduling for Prefill Optimization
Develop a dynamic execution engine with a hybrid scheduler that switches between parallel GEMM kernels for prefill and ELUT kernels for decoding.
Source paper
Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, Furu Wei
Generated by AI. The paper’s authors did not write these.
Develop a dynamic execution engine with a hybrid scheduler that switches between parallel GEMM kernels for prefill and ELUT kernels for decoding.
Design a custom hardware accelerator or ISA extension implementing a fused lookup-accumulate instruction to reduce ELUT latency.
Port element-wise LUTs and signed-unsigned weight splitting logic to CUDA or OpenAI Triton for execution on many-core GPU architectures.
Refactor the kernel generator to support arbitrary alphabet sizes and group sizes for universal non-standard low-bit quantization inference.
This paper introduces Bitnet.cpp, an inference library optimized for ternary (1.58-bit) Large Language Models (LLMs) like BitNet b1.58 on edge CPUs. It proposes a novel mixed-precision matrix multiplication (mpGEMM) library featuring Element-wise Lookup Table (TL) and "Int2 with a Scale" (I2_S) kernels to resolve spatial inefficiencies in existing bit-wise methods and achieve fast, lossless inference.
The authors implement a custom mpGEMM library that replaces bit-wise operations with element-wise processing. Key techniques include "Element-wise Mirror Consolidation" to compress ternary weights into 1.67 bits/weight (packing 3 weights into 5 bits) using signed-unsigned splitting, and an I2_S kernel that strictly aligns with the specific per-tensor quantization scheme of BitNet b1.58 training to ensure zero accuracy loss.