Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Distilling Open-Ended Generation from RandOpt Ensembles

Develop a sequence-level knowledge distillation framework that aligns student output distributions with averaged token-level probability distributions from an ensemble of teachers.

Time

3 hours

Compute level

Large cluster

Resources

1 node of 8x NVIDIA GH200 GPUs for 1 hour to perform generation from K teachers and subsequent SFT distillation.

What the paper used

Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct models using a population of 5,000 models, top-50 models selected for distillation over 10 SGD iterations using NVIDIA GH200 GPUs.

Prerequisites

transformer architectures

Scope

modify existing code

The Gap

The distillation approach described in Section 7 is explicitly tailored to LLM reasoning tasks that end in a categorical final prediction (e.g., math answers), making it difficult to apply to code generation or creative writing tasks.

“A catch with RandOpt, compared to standard post-training, is that good performance requires K forward passes at test time. To address this, we explore distilling the top-K models into a single model.”
From the paper, Section 7, page 10. Matched against the PDF text.

Limitation

RandOpt incurs a high inference-time cost by requiring K forward passes, and the paper's proposed distillation fix cannot currently handle open-ended text where strict majority voting on identical output sequences is impossible.

Proposed Solution

Develop a sequence-level knowledge distillation framework that aligns the student model's output distribution with the averaged token-level probability distributions of the K teacher models (the RandOpt ensemble) at each autoregressive decoding step, rather than relying on exact trace matches.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

This would allow the performance benefits of RandOpt to be deployed efficiently at inference time for a much broader range of generative tasks, drastically reducing latency while maintaining ensemble-level reasoning.

How to Get Started

How to test it: Hypothesis: Aligning the student model to the averaged token-level probability distribution of the K teacher models achieves performance close to the ensemble. Baseline: The un-distilled base model or discrete trace-matching distillation. Metric: Accuracy on GSM8K and MATH-500. Stopping condition: If the distilled student achieves at least 95% of the ensemble's accuracy while requiring only a single forward pass.

  1. Set up an ensemble of diverse teacher models to generate multiple reasoning traces for open-ended code generation and creative writing tasks.
  2. Compute averaged token-level probability distributions across the ensemble outputs during autoregressive generation steps.
  3. Implement a sequence-level divergence loss function to align the student model's token distribution with the ensemble average.
  4. Evaluate the distilled student model on standard open-ended text and code generation benchmarks.

Code linked to this paper on Hugging Face: github.com/sunrainyg/RandOpt

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

Papers like Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties and Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching discuss ensemble distillation and logit matching for language models. However, none of the listed works specifically propose distilling an open-ended RandOpt ensemble using token-level probability averaging at each autoregressive step.

We searched OpenAlex and arXiv for “Develop a sequence-level knowledge distillation framework that aligns student output distributions with averaged token-level probability distributions from an ensemble of teachers.” and “Distilling Open-Ended Generation from RandOpt Ensembles” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan, Phillip Isola · 2026

This paper reveals that large pretrained models exist in a 'thicket' regime, where the local neighborhood around their weights is densely populated with diverse, task-specific experts. Based on this finding, the authors introduce RandOpt, a gradient-free post-training method that samples random weight perturbations and ensembles the best ones via majority voting to achieve competitive performance against standard RL methods.

Related Ideas

Is this a good idea?

Share onXLinkedIn