Length-Extrapolation via Curriculum Learning
Extend maximum sequence length capabilities and reduce hallucinations by fine-tuning models on progressively longer audio clips using a curriculum learning strategy.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 4.1 notes a notable increase in hallucinations for clips exceeding the 30-second sequence length observed during training. The model fails to generalize RoPE embeddings to durations it hasn't seen explicitly.
“Conversely, for longer clips—particularly those exceeding the sequence lengths observed during training—there is a notable increase in transcription errors due to hallucinations. To evaluate this behavior, we measured the WER across a range of audio segments from the TEDLium dataset, with clip durations varying between 10 and 55 seconds.”
Limitation
The model's reliability degrades sharply for audio inputs longer than the maximum training duration (30s).
Proposed Solution
Implement a curriculum learning strategy where the model is fine-tuned on progressively longer sequences (e.g., up to 60 or 90 seconds). Combine this with linear attention mechanisms or sliding window attention to prevent the quadratic compute cost from exploding at these longer lengths.
Why This Matters
Enables the model to handle long-form content (like lectures) in a single pass without arbitrary chunking, reducing context fragmentation.
How to Get Started
How to test it: We hypothesize that curriculum learning on up to 90-second sequences will decrease hallucination rates on long audio clips compared to the baseline. We will evaluate WER on clips >30 seconds, stopping if the hallucination rate does not drop below 5% on 60-second clips.
- Collect long-form audio datasets exceeding thirty seconds and split them into incremental duration buckets.
- Implement a sliding window attention mechanism to manage computational overhead during extended sequence processing.
- Fine-tune the baseline model sequentially across increasing length brackets while monitoring hallucination rates on validation clips.
Code linked to this paper on Hugging Face: github.com/usefulsensors/moonshine
Suggested by AI. Adapt it to your setup.
Has This Been Done?
While the search results include various audio and multimodal foundation models like Audio Flamingo 2 as well as unrelated curriculum learning papers, none of them evaluate a curriculum learning strategy paired with linear or sliding window attention specifically for length-extrapolation in speech models.
We searched OpenAlex for “Extend maximum sequence length capabilities and reduce hallucinations by fine-tuning models on progressively longer audio clips using a curriculum learning strategy.” and “Length-Extrapolation via Curriculum Learning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Moonshine: Speech Recognition for Live Transcription and Voice CommandsNat Jeffries, Evan King, Manjunath Kudlur +3 more · 2024
This paper introduces Moonshine, a family of lightweight speech recognition models designed to overcome the fixed-latency limitations of OpenAI's Whisper by using variable-length encoding. By replacing absolute position embeddings with Rotary Position Embeddings (RoPE) and eliminating zero-padding, Moonshine achieves up to 5x faster inference on short audio segments while maintaining comparable Word Error Rates (WER) to Whisper tiny.en.
Related Ideas
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Native Long-Context Modeling via Ring AttentionSAM Audio: Segment Anything in Audio🔴Hard
- Infinite-Context Dual-Autoregressive Architecture for Unbounded Audiobook SynthesisFish Audio S2 Technical Report🔴Hard
- Dynamic Attention-Based Identity RoutingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟡Fairly easy
- Recurrent Visual Memory for Infinite-Context TrajectoriesMobile-Agent-v3: Foundamental Agents for GUI Automation🔴Hard