Low-Resource Language Adaptation via Parameter-Efficient Fine-Tuning in Non-G2P Models
Deploy low-rank adaptation layers on the Slow Transformer cross-attention modules to improve low-resource language speech adaptation without catastrophic forgetting.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The model's training dataset (Section 5) is heavily skewed, containing 600,000 hours of English and Mandarin, but only 20,000 hours for each of the other language families (Germanic, Romance, East Asian, Semitic). The G2P-free, pure LLM-based feature extraction approach heavily relies on massive data to implicitly learn phonetic rules.
“The dataset contains about 720,000 hours of speech across different languages, with 300,000 hours each of English and Mandarin Chinese as the main components.”
Limitation
The massive data imbalance likely causes the LLM linguistic feature extraction to underperform or hallucinate phonetic structures when processing low-resource languages, due to a lack of sufficient implicit acoustic-semantic mapping.
Proposed Solution
Implement Parameter-Efficient Fine-Tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA), specifically tailored to the Slow Transformer's cross-attention layers. Train these language-specific LoRA modules exclusively on low-resource datasets, effectively creating modular language adapters that stabilize codebook generation without causing catastrophic forgetting of the dominant languages.
Why This Matters
This addresses the scalability and equity of multilingual TTS, proving that non-G2P frameworks can be viably adapted to low-resource languages without needing hundreds of thousands of hours of parallel data.
How to Get Started
How to test it: We hypothesize that LoRA adapters on cross-attention layers improve low-resource phonetic mapping without catastrophic forgetting. We measure phoneme error rate on low-resource test sets compared to full fine-tuning, stopping if dominant language WER degrades by more than 1%.
- Extract public low-resource speech datasets such as Common Voice for target language families.
- Attach Low-Rank Adaptation modules specifically to the cross-attention layers of the Slow Transformer.
- Fine-tune the LoRA weights using low-resource speech data while keeping the base model parameters frozen.
- Evaluate codebook generation stability and word error rates across both high-resource and low-resource languages.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the retrieved sources specifically investigate applying LoRA specifically to the cross-attention layers of a Slow Transformer for low-resource language adaptation in non-G2P models. Although there are general reviews on LoRA and papers covering low-resource speech adaptation such as Robust Low-Resource Dysarthric Speech Recognition via Low-Rank Adaptation, the specific architectural adaptation proposed remains unexplored by these results.
- Robust Low-Resource Dysarthric Speech Recognition via Low-Rank Adaptation · 2026 · 0 citations
- Low-Rank Adaptation for Foundation Models: A Comprehensive Review · 2025 · 105 citations
- Text-Inductive Graphone-Based Language Adaptation for Low-Resource Speech Synthesis · 2024 · 16 citations
We searched OpenAlex and arXiv for “Deploy low-rank adaptation layers on the Slow Transformer cross-attention modules to improve low-resource language speech adaptation without catastrophic forgetting.” and “Low-Resource Language Adaptation via Parameter-Efficient Fine-Tuning in Non-G2P Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech SynthesisShijia Liao, Yuxuan Wang, Tianyu Li +4 more · 2024
This paper introduces Fish-Speech, a novel multilingual Text-to-Speech (TTS) framework that leverages Large Language Models (LLMs) to bypass traditional grapheme-to-phoneme conversions. Its primary contribution is enhancing sequence generation stability and multilingual voice cloning capabilities through a dual autoregressive architecture and an advanced Grouped Finite Scalar Vector Quantization vocoder.
Related Ideas
- Cross-Lingual Timbre Transfer for Low-Resource TTS AlignmentFish Audio S2 Technical Report🟠Medium
- Phoneme-Aware LoRA Adapters for Low-Resource LanguagesLTX-2: Efficient Joint Audio-Visual Foundation Model🟡Fairly easy
- Cross-Lingual Phonetic Transfer for Under-Resourced ASRFunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs🟠Medium
- Optimizing Parameter Efficiency via Language-Specific TokenizationFlavors of Moonshine: Tiny Specialized ASR Models for Edge Devices🟡Fairly easy
- Synthetic Cross-Lingual Speaker AdaptationQwen3-TTS Technical Report🟢Easy