MIMO-Vibe: Multi-Stream Latent Diffusion for Overlapping Speech
Extend audio generation models to a multi-input multi-output framework with joint latent denoising to synthesize overlapping speech segments.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The Input Representation described in Section 2.2 utilizes a strictly serial concatenation of speaker tokens and text scripts, and the model conclusion confirms it cannot generate overlapping speech segments.
Limitation
Natural human conversation frequently involves interruptions and backchanneling, which the current sequential autoregressive modeling cannot represent or synthesize.
Proposed Solution
Extend the architecture to a Multi-Input Multi-Output (MIMO) framework where the LLM predicts hidden states for parallel audio tracks. This involves modifying the diffusion head to perform joint denoising on a dual-latent space, allowing the model to generate two simultaneous waveforms when the text script indicates overlapping dialogue.
Why This Matters
Enables the synthesis of truly realistic dialogues, podcasts, and debates where turn-taking is fluid and non-linear.
How to Get Started
- Prepare overlapping speech benchmarks using LibriCSS or synthetic mixing pipelines.
- Modify the underlying LLM to output parallel hidden states for simultaneous audio tracks.
- Adapt the diffusion head to perform joint denoising on a dual-latent audio space.
- Train the model on multi-speaker datasets and evaluate separation and synthesis quality.
Code linked to this paper on Hugging Face: github.com/microsoft/VibeVoice
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the provided papers discuss a multi-stream latent diffusion framework using MIMO for overlapping speech dialogue generation. While works like SepGen and MMControl deal with multi-stem audio or joint audio-video generation, they do not specifically implement a dual-latent space for overlapping speech prediction via LLM hidden states.
- MMControl: Unified Multi-modal Control for Joint Audio-Video Generation · 2026 · 6 citations
- SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model · 2026 · 0 citations
- MMEDIT: A Unified Framework for Multi-Type Audio Editing via Audio Language Model · 2025 · 16 citations
We searched OpenAlex for “Extend audio generation models to a multi-input multi-output framework with joint latent denoising to synthesize overlapping speech segments.” and “MIMO-Vibe: Multi-Stream Latent Diffusion for Overlapping Speech” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
VibeVoice Technical ReportZhiliang Peng, Jianwei Yu, Wenhui Wang +10 more · 2025
VIBEVOICE is a novel speech synthesis framework designed for generating long-form, multi-speaker conversational audio, supporting sequences up to 90 minutes. It introduces an ultra-high 3200x compression causal speech tokenizer that enables efficient processing of long sequences within a Large Language Model (LLM) framework using next-token diffusion.
Related Ideas
- End-to-End Speech-to-Speech Integration via Latent S3 Token AlignmentFunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs🔴Hard
- Data Synthesis for Backchannel ModelingOmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation🟡Fairly easy
- Hierarchical Disentanglement for Controllable TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🟠Medium
- Homogeneous Mamba-Based Audio TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🔴Hard
- Native Long-Context Modeling via Ring AttentionSAM Audio: Segment Anything in Audio🔴Hard