Distillation-Enhanced ARLM-to-DLM Conversion
A method that aligns autoregressive language models to diffusion frameworks using logit distillation to preserve the teacher model's probabilistic priors and prevent severe performance regressions.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 4.2.2 and Table 3, the authors show that converting Autoregressive LMs (ARLMs) like Qwen to DLMs via SFT leads to significant performance regression (e.g., MMLU score drops from 52.8 to 40.0). The authors attribute this to the lack of continual pretraining, but the SFT-only approach fails to fully leverage the teacher's existing capabilities.
“Despite their traditional use in discriminative tasks, BERT-style models (Devlin et al., 2019) offer bidirectional representations well-suited for diffusive generation (Sahoo et al., 2024).”
Limitation
The current conversion recipe (SFT only) is lossy, failing to preserve the full reasoning and knowledge capabilities of the autoregressive backbone.
Proposed Solution
Implement a Logit Distillation Trainer within dLLM. Instead of standard cross-entropy on ground truth, the diffusion model should be trained to match the output distribution of the frozen autoregressive teacher (appropriately mapped to the diffusion state). This would force the DLM to retain the teacher's probabilistic priors more effectively than hard-label SFT.
Why This Matters
Closing the performance gap between ARLMs and their converted DLM counterparts is crucial for making diffusion models a viable, competitive alternative for text generation.
How to Get Started
How to test it: Hypothesis: Logit distillation from an autoregressive teacher preserves reasoning capabilities better than hard-label SFT. Baseline: Standard cross-entropy SFT on the same architecture. Metric: MMLU and GSM8K accuracy. Stopping condition: If distillation accuracy falls below hard-label SFT baseline by epoch 5.
- Set up the baseline dLLM training environment and reproduce the standard supervised fine-tuning performance drop on smaller benchmarks.
- Implement a logit mapping function that aligns the discrete vocabulary outputs of an autoregressive teacher like Qwen with the continuous diffusion states.
- Write a custom training loss combining cross-entropy with Kullback-Leibler divergence matching against the frozen teacher's output distribution.
- Fine-tune a mid-sized diffusion model using the distillation loss and evaluate on MMLU to measure score retention compared to hard-label training.
Code linked to this paper on Hugging Face: github.com/ZHZisZZ/dllm
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation explores the conversion of autoregressive models to diffusion language models using distillation techniques.
- Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation · 2026 · 13 citations
We searched OpenAlex for “A method that aligns autoregressive language models to diffusion frameworks using logit distillation to preserve the teacher model's probabilistic priors and prevent severe performance regressions.” and “Distillation-Enhanced ARLM-to-DLM Conversion” and an AI compared the idea with the top 7 results. A small search can miss relevant work.
Source Paper
dLLM: Simple Diffusion Language ModelingZhanhui Zhou, Lingjie Chen, Hanghang Tong +1 more · 2026
This paper introduces dLLM, a unified, open-source framework designed to standardize the training, inference, and evaluation of Diffusion Language Models (DLMs). It provides modular components for reproducing existing models (like LLaDA and Dream), recipes for converting autoregressive models and BERT encoders into DLMs with minimal compute, and a comprehensive evaluation pipeline to benchmark performance and speed.
Related Ideas
- Distilling Open-Ended Generation from RandOpt EnsemblesNeural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights🟡Fairly easy
- Automated Operator Distillation for Cost-Efficient ScalingDataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI🟡Fairly easy
- Decoupled Text Representation DistillationMobile-O: Unified Multimodal Understanding and Generation on Mobile Device🟠Medium
- Information-Theoretic Quantification of CFG-Induced Path CollapseDecoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield🔴Hard
- Uncertainty-Aware Robust DistillationLiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation🟡Fairly easy