Skip to content
AI-generated research idea🔴Hard💡Moderate InnovationNo close match found

Length-Extrapolation via Curriculum Learning

Extend maximum sequence length capabilities and reduce hallucinations by fine-tuning models on progressively longer audio clips using a curriculum learning strategy.

Time

3 days

Compute level

Large cluster

Resources

32x H100 GPU cluster for sequence-length extension training.

What the paper used

Trained on a 32x H100 GPU cluster for 250K steps with a global batch size of 1024.

Prerequisites

transformer architectures, curriculum learning

Scope

modify existing code

The Gap

Section 4.1 notes a notable increase in hallucinations for clips exceeding the 30-second sequence length observed during training. The model fails to generalize RoPE embeddings to durations it hasn't seen explicitly.

“Conversely, for longer clips—particularly those exceeding the sequence lengths observed during training—there is a notable increase in transcription errors due to hallucinations. To evaluate this behavior, we measured the WER across a range of audio segments from the TEDLium dataset, with clip durations varying between 10 and 55 seconds.”
From the paper, Page 5, Section 4.1. Matched against the PDF text.

Limitation

The model's reliability degrades sharply for audio inputs longer than the maximum training duration (30s).

Proposed Solution

Implement a curriculum learning strategy where the model is fine-tuned on progressively longer sequences (e.g., up to 60 or 90 seconds). Combine this with linear attention mechanisms or sliding window attention to prevent the quadratic compute cost from exploding at these longer lengths.

Why This Matters

Enables the model to handle long-form content (like lectures) in a single pass without arbitrary chunking, reducing context fragmentation.

How to Get Started

How to test it: We hypothesize that curriculum learning on up to 90-second sequences will decrease hallucination rates on long audio clips compared to the baseline. We will evaluate WER on clips >30 seconds, stopping if the hallucination rate does not drop below 5% on 60-second clips.

  1. Collect long-form audio datasets exceeding thirty seconds and split them into incremental duration buckets.
  2. Implement a sliding window attention mechanism to manage computational overhead during extended sequence processing.
  3. Fine-tune the baseline model sequentially across increasing length brackets while monitoring hallucination rates on validation clips.

Code linked to this paper on Hugging Face: github.com/usefulsensors/moonshine

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

While the search results include various audio and multimodal foundation models like Audio Flamingo 2 as well as unrelated curriculum learning papers, none of them evaluate a curriculum learning strategy paired with linear or sliding window attention specifically for length-extrapolation in speech models.

We searched OpenAlex for “Extend maximum sequence length capabilities and reduce hallucinations by fine-tuning models on progressively longer audio clips using a curriculum learning strategy.” and “Length-Extrapolation via Curriculum Learning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Moonshine: Speech Recognition for Live Transcription and Voice Commands

Nat Jeffries, Evan King, Manjunath Kudlur +3 more · 2024

This paper introduces Moonshine, a family of lightweight speech recognition models designed to overcome the fixed-latency limitations of OpenAI's Whisper by using variable-length encoding. By replacing absolute position embeddings with Rotary Position Embeddings (RoPE) and eliminating zero-padding, Moonshine achieves up to 5x faster inference on short audio segments while maintaining comparable Word Error Rates (WER) to Whisper tiny.en.

Related Ideas

Is this a good idea?

Share onXLinkedIn