Skip to content

Source paper

Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis

Shijia Liao, Yuxuan Wang, Tianyu Li and 4 more

Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, Yijin Xing

arXiv 2411.01156 · 202413 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces Fish-Speech, a novel multilingual Text-to-Speech (TTS) framework that leverages Large Language Models (LLMs) to bypass traditional grapheme-to-phoneme conversions. Its primary contribution is enhancing sequence generation stability and multilingual voice cloning capabilities through a dual autoregressive architecture and an advanced Grouped Finite Scalar Vector Quantization vocoder.

How it works (AI summary)

The system utilizes a Dual Autoregressive (Dual-AR) architecture consisting of a Slow Transformer for global semantic token generation and a Fast Transformer for detailed acoustic codebook processing. It is paired with FF-GAN, an enhanced vocoder that employs Grouped Finite Scalar Vector Quantization (GFSQ) and ParallelBlock convolutions to achieve near 100% codebook utilization and high-fidelity audio reconstruction.