Skip to content

Source paper

Bitnet.cpp: Efficient Edge Inference for Ternary LLMs

Jinheng Wang, Hansong Zhou, Ting Song and 7 more

Jinheng Wang, Hansong Zhou, Ting Song, Shijie Cao, Yan Xia, Ting Cao, Jianyu Wei, Shuming Ma, Hongyu Wang, Furu Wei

arXiv 2502.11880 · 20252 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces Bitnet.cpp, an inference library optimized for ternary (1.58-bit) Large Language Models (LLMs) like BitNet b1.58 on edge CPUs. It proposes a novel mixed-precision matrix multiplication (mpGEMM) library featuring Element-wise Lookup Table (TL) and "Int2 with a Scale" (I2_S) kernels to resolve spatial inefficiencies in existing bit-wise methods and achieve fast, lossless inference.

How it works (AI summary)

The authors implement a custom mpGEMM library that replaces bit-wise operations with element-wise processing. Key techniques include "Element-wise Mirror Consolidation" to compress ternary weights into 1.67 bits/weight (packing 3 weights into 5 bits) using signed-unsigned splitting, and an I2_S kernel that strictly aligns with the specific per-tensor quantization scheme of BitNet b1.58 training to ensure zero accuracy loss.