Skip to content

Source paper

BitNet Distillation

Xun Wu, Shaohan Huang, Wenhui Wang and 4 more

Xun Wu, Shaohan Huang, Wenhui Wang, Ting Song, Li Dong, Yan Xia, Furu Wei

arXiv 2510.13998 · 202555 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

AI summary

The paper introduces BitNet Distillation (BitDistill), a framework designed to fine-tune existing full-precision Large Language Models (LLMs) into 1.58-bit ternary models for specific downstream tasks without expensive training from scratch. This approach aims to achieve performance comparable to full-precision counterparts while significantly reducing memory usage and inference latency.

How it works (AI summary)

The method employs a three-stage pipeline: (1) Architectural refinement by inserting `SubLN` (Subject Layer Normalization) modules to stabilize training; (2) A 'Continue Pre-training' phase on a small corpus to adapt weights to the ternary representation and mitigate scalability issues; and (3) A distillation process utilizing a fine-tuned full-precision teacher, incorporating both logits distillation and a selective Multi-Head Attention distillation.