Skip to content

Source paper

OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation

Qinglin Zhang, Luyao Cheng, Chong Deng and 6 more

Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan

arXiv 2410.17799 · 20249 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces OmniFlatten, an end-to-end GPT-based model designed for full-duplex spoken dialogue, enabling simultaneous listening and speaking (barge-in). The authors propose a multi-stage post-training scheme that progressively adapts a text-based LLM to handle speech and text streams via a flattening operation, avoiding architectural modifications to the backbone.

How it works (AI summary)

The model uses a standard Transformer decoder (Qwen2-0.5B) where speech inputs/outputs are discretized into tokens using a CosyVoice tokenizer. These speech tokens are interleaved with text tokens into a single flattened sequence. The training involves three stages: Modality Alignment (ASR/TTS tasks), Half-duplex Dialogue (turn-based), and Full-duplex Dialogue (simultaneous streams using chunked data simulation).