Mitigating the Modality Gap via Mixture-of-Experts (MoE) Routing
Implement a sparse Mixture-of-Experts routing architecture within language model transformer blocks to preserve text reasoning while handling parallel acoustic features.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In the Spoken Language Modeling evaluation on benchmarks like sSC and tSC (Section 6.3), the model shows a drop in text-only accuracy compared to the base Llama model.
“On sSC and tSC, we observe an accuracy drop when comparing TADA operating in text-only mode to the base Llama models, suggesting the model trades off some of its text pretraining capacity to accommodate the speech modality.”
Limitation
Training the unified model on text-speech data causes a degradation in pure text-only reasoning capacity, indicating that the LLM trades off some of its foundational linguistic pretraining to accommodate the new speech modality.
Proposed Solution
Implement a sparse Mixture-of-Experts (MoE) routing architecture within the LLM transformer blocks. Dedicated text-only experts can be frozen or regularized to preserve base reasoning, while new speech-aware experts are dynamically routed to handle the parallel acoustic feature modeling.
Why This Matters
This prevents catastrophic forgetting of pure language intelligence, allowing unified multimodal systems to scale speech capabilities without sacrificing state-of-the-art text reasoning.
How to Get Started
How to test it: Hypothesis: Converting Llama 3.2 1B/3B FFN blocks into sparse MoE layers with frozen text experts and trainable speech-routed experts will eliminate text-only capability degradation while preserving multimodal quality. Compare against base Llama-1B/3B-Instruct and TADA-1B/3B-ML (with SFG) on SI PPL, sSC, tSC, CER, SIM, and oMOS, stopping if text-only sSC/tSC drops noticeably below base Llama-3B-Instruct (79.4/98.0) or text-speech sSC/tSC falls below TADA-3B-ML w/ SFG (66.7/94.7).
- Download the base Llama model and inspect its transformer block definitions.
- Implement a routing mechanism that splits incoming tokens between frozen text experts and newly introduced speech experts.
- Set up the sSC and tSC evaluation benchmarks to measure text-only accuracy alongside speech modeling performance.
- Train the modified model on a combined speech-text dataset while regularizing the base experts against catastrophic forgetting.
Code linked to this paper on Hugging Face: github.com/HumeAI/tada
Suggested by AI. Adapt it to your setup.
Has This Been Done?
MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts directly targets mixing speech and text using modality-aware mixture-of-experts architectures. Additionally, Uni-MoE and MoE-LLaVA explore scaling multimodal LLMs with mixture of experts to bridge modality gaps.
- Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts · 2025 · 252 citations
- MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts · 2026 · 2 citations
- MoE-LLaVA: Mixture of Experts for Large Vision-Language Models · 2026 · 542 citations
We searched OpenAlex and arXiv for “Implement a sparse Mixture-of-Experts routing architecture within language model transformer blocks to preserve text reasoning while handling parallel acoustic features.” and “Mitigating the Modality Gap via Mixture-of-Experts (MoE) Routing” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual AlignmentTrung Dang, Sharath Rao, Ananya Gupta +6 more · 2026
This paper introduces TADA, a generative framework that unifies text and speech modeling within a Large Language Model by establishing a one-to-one synchronization between continuous acoustic features and discrete text tokens. This approach addresses the frequency mismatch between text and audio, virtually eliminating content hallucinations and enabling highly efficient, single-stream autoregressive speech synthesis.
Related Ideas
- Sparse Scaling via Multimodal Mixture-of-ExpertsNextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation🟠Medium
- End-to-End Speech-to-Speech Integration via Latent S3 Token AlignmentFunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs🔴Hard
- Low-Resource Language Adaptation via Parameter-Efficient Fine-Tuning in Non-G2P ModelsFish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis🟡Fairly easy
- Adaptive Token-Level Gating for Heterogeneous Modality AlignmentInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🟡Fairly easy
- End-to-End Joint Optimization of MLLM and MMDiTUniVideo: Unified Understanding, Generation, and Editing for Videos🔴Hard