Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchNo close match found

MIMO-Vibe: Multi-Stream Latent Diffusion for Overlapping Speech

Extend audio generation models to a multi-input multi-output framework with joint latent denoising to synthesize overlapping speech segments.

Time

3 to 6 months

Compute level

Multiple GPUs

Resources

High-end multi-GPU setup (e.g., 4-8x A100s), datasets containing overlapping speech (e.g., LibriCSS, AMI) or synthetic mixing pipelines

Prerequisites

Advanced understanding of Latent Diffusion Models (LDM), Transformer architectures, and Audio Signal Processing

Scope

Significant architectural modification (redesigning output heads for MIMO), custom data loader for parallel tracks, modified loss function for joint denoising

The Gap

The Input Representation described in Section 2.2 utilizes a strictly serial concatenation of speaker tokens and text scripts, and the model conclusion confirms it cannot generate overlapping speech segments.

Limitation

Natural human conversation frequently involves interruptions and backchanneling, which the current sequential autoregressive modeling cannot represent or synthesize.

Proposed Solution

Extend the architecture to a Multi-Input Multi-Output (MIMO) framework where the LLM predicts hidden states for parallel audio tracks. This involves modifying the diffusion head to perform joint denoising on a dual-latent space, allowing the model to generate two simultaneous waveforms when the text script indicates overlapping dialogue.

Why This Matters

Enables the synthesis of truly realistic dialogues, podcasts, and debates where turn-taking is fluid and non-linear.

How to Get Started

  1. Prepare overlapping speech benchmarks using LibriCSS or synthetic mixing pipelines.
  2. Modify the underlying LLM to output parallel hidden states for simultaneous audio tracks.
  3. Adapt the diffusion head to perform joint denoising on a dual-latent audio space.
  4. Train the model on multi-speaker datasets and evaluate separation and synthesis quality.

Code linked to this paper on Hugging Face: github.com/microsoft/VibeVoice

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

None of the provided papers discuss a multi-stream latent diffusion framework using MIMO for overlapping speech dialogue generation. While works like SepGen and MMControl deal with multi-stem audio or joint audio-video generation, they do not specifically implement a dual-latent space for overlapping speech prediction via LLM hidden states.

We searched OpenAlex for “Extend audio generation models to a multi-input multi-output framework with joint latent denoising to synthesize overlapping speech segments.” and “MIMO-Vibe: Multi-Stream Latent Diffusion for Overlapping Speech” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

VibeVoice Technical Report

Zhiliang Peng, Jianwei Yu, Wenhui Wang +10 more · 2025

VIBEVOICE is a novel speech synthesis framework designed for generating long-form, multi-speaker conversational audio, supporting sequences up to 90 minutes. It introduces an ultra-high 3200x compression causal speech tokenizer that enables efficient processing of long sequences within a Large Language Model (LLM) framework using next-token diffusion.

Related Ideas

Is this a good idea?

Share onXLinkedIn