Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchClosely matching work found

Distillation-Enhanced ARLM-to-DLM Conversion

A method that aligns autoregressive language models to diffusion frameworks using logit distillation to preserve the teacher model's probabilistic priors and prevent severe performance regressions.

Time

1 week

Compute level

Multiple GPUs

Resources

8x A100 GPUs with DeepSpeed ZeRO-2 for teacher-student logit matching.

What the paper used

8x A100 GPUs with DeepSpeed ZeRO-2

Prerequisites

transformer architectures

Scope

modify existing code

The Gap

In Section 4.2.2 and Table 3, the authors show that converting Autoregressive LMs (ARLMs) like Qwen to DLMs via SFT leads to significant performance regression (e.g., MMLU score drops from 52.8 to 40.0). The authors attribute this to the lack of continual pretraining, but the SFT-only approach fails to fully leverage the teacher's existing capabilities.

“Despite their traditional use in discriminative tasks, BERT-style models (Devlin et al., 2019) offer bidirectional representations well-suited for diffusive generation (Sahoo et al., 2024).”
From the paper, Section 4.2.1, page 7. Matched against the PDF text.

Limitation

The current conversion recipe (SFT only) is lossy, failing to preserve the full reasoning and knowledge capabilities of the autoregressive backbone.

Proposed Solution

Implement a Logit Distillation Trainer within dLLM. Instead of standard cross-entropy on ground truth, the diffusion model should be trained to match the output distribution of the frozen autoregressive teacher (appropriately mapped to the diffusion state). This would force the DLM to retain the teacher's probabilistic priors more effectively than hard-label SFT.

Why This Matters

Closing the performance gap between ARLMs and their converted DLM counterparts is crucial for making diffusion models a viable, competitive alternative for text generation.

How to Get Started

How to test it: Hypothesis: Logit distillation from an autoregressive teacher preserves reasoning capabilities better than hard-label SFT. Baseline: Standard cross-entropy SFT on the same architecture. Metric: MMLU and GSM8K accuracy. Stopping condition: If distillation accuracy falls below hard-label SFT baseline by epoch 5.

  1. Set up the baseline dLLM training environment and reproduce the standard supervised fine-tuning performance drop on smaller benchmarks.
  2. Implement a logit mapping function that aligns the discrete vocabulary outputs of an autoregressive teacher like Qwen with the continuous diffusion states.
  3. Write a custom training loss combining cross-entropy with Kullback-Leibler divergence matching against the frozen teacher's output distribution.
  4. Fine-tune a mid-sized diffusion model using the distillation loss and evaluate on MMLU to measure score retention compared to hard-label training.

Code linked to this paper on Hugging Face: github.com/ZHZisZZ/dllm

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Closely matching work found

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation explores the conversion of autoregressive models to diffusion language models using distillation techniques.

We searched OpenAlex for “A method that aligns autoregressive language models to diffusion frameworks using logit distillation to preserve the teacher model's probabilistic priors and prevent severe performance regressions.” and “Distillation-Enhanced ARLM-to-DLM Conversion” and an AI compared the idea with the top 7 results. A small search can miss relevant work.

Source Paper

dLLM: Simple Diffusion Language Modeling

Zhanhui Zhou, Lingjie Chen, Hanghang Tong +1 more · 2026

This paper introduces dLLM, a unified, open-source framework designed to standardize the training, inference, and evaluation of Diffusion Language Models (DLMs). It provides modular components for reproducing existing models (like LLaDA and Dream), recipes for converting autoregressive models and BERT encoders into DLMs with minimal compute, and a comprehensive evaluation pipeline to benchmark performance and speed.

Related Ideas

Is this a good idea?

Share onXLinkedIn