Scaling Backbone for Semantic Robustness
A scalable voice-chat backbone system that replaces small foundational models with larger open-weights models to improve conversational reasoning and knowledge.
Source paper
Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chaohong Tan
Generated by AI. The paper’s authors did not write these.
A scalable voice-chat backbone system that replaces small foundational models with larger open-weights models to improve conversational reasoning and knowledge.
A data synthesis pipeline that overlays non-interruptive backchannel speech tokens onto training audio streams to teach models conversational listening behaviors.
Modify the dialogue learning loss function to reduce penalties for missing text alignment when an interruption token is predicted, allowing the model to prioritize acoustic stop actions.
This paper introduces OmniFlatten, an end-to-end GPT-based model designed for full-duplex spoken dialogue, enabling simultaneous listening and speaking (barge-in). The authors propose a multi-stage post-training scheme that progressively adapts a text-based LLM to handle speech and text streams via a flattening operation, avoiding architectural modifications to the backbone.
The model uses a standard Transformer decoder (Qwen2-0.5B) where speech inputs/outputs are discretized into tokens using a CosyVoice tokenizer. These speech tokens are interleaved with text tokens into a single flattened sequence. The training involves three stages: Modality Alignment (ASR/TTS tasks), Half-duplex Dialogue (turn-based), and Full-duplex Dialogue (simultaneous streams using chunked data simulation).