Automated Sensitivity-Based Layer Selection
Develop an automated, training-free metric based on Hessian sensitivity or Cakaran analysis to rank attention layers for optimal distillation.
Source paper
Xun Wu, Shaohan Huang, Wenhui Wang, Ting Song, Li Dong, Yan Xia, Furu Wei
Generated by AI. The paper’s authors did not write these.
Develop an automated, training-free metric based on Hessian sensitivity or Cakaran analysis to rank attention layers for optimal distillation.
Adapt distillation loss functions to incorporate Chain-of-Thought reasoning steps and penalize student divergence on complex reasoning tasks.
Align domain adaptation by replacing generic web text corpora with self-supervised language modeling on the unlabeled target dataset during pre-training.
Extend ternary quantization frameworks to support 4-bit activations by integrating optimal clipping threshold learning into the SubLN module.
The paper introduces BitNet Distillation (BitDistill), a framework designed to fine-tune existing full-precision Large Language Models (LLMs) into 1.58-bit ternary models for specific downstream tasks without expensive training from scratch. This approach aims to achieve performance comparable to full-precision counterparts while significantly reducing memory usage and inference latency.
The method employs a three-stage pipeline: (1) Architectural refinement by inserting `SubLN` (Subject Layer Normalization) modules to stabilize training; (2) A 'Continue Pre-training' phase on a small corpus to adapt weights to the ternary representation and mitigate scalability issues; and (3) A distillation process utilizing a fine-tuned full-precision teacher, incorporating both logits distillation and a selective Multi-Head Attention distillation.