Semantic-Driven Prosody Inference for Implicit Emotion Control
Map LLM semantic hidden states directly into speech synthesis style spaces to automate expressive emotional tone generation without explicit tags.
Source paper
Generated by AI. The paper’s authors did not write these.
Map LLM semantic hidden states directly into speech synthesis style spaces to automate expressive emotional tone generation without explicit tags.
Replace modular speech-to-text-to-speech pipelines with unified architectures that feed acoustic speech tokens directly into language model embeddings.
Enable real-time streaming speech recognition by modifying memory-equipped self-attention encoders with causal chunk-based masking and Transducer losses.
Transfer universal phonetic representations to under-resourced languages using parameter-efficient fine-tuning on language-agnostic features.
FunAudioLLM is a foundation model family comprising SenseVoice for multilingual speech understanding and CosyVoice for natural voice generation. These models aim to bridge the gap between human voice interaction and large language models (LLMs) by providing high-speed ASR, emotion recognition, and zero-shot voice cloning capabilities.
The framework utilizes a non-autoregressive SAN-M architecture for low-latency speech recognition (SenseVoice-Small) and an autoregressive Transformer encoder-decoder for high-precision multilingual tasks (SenseVoice-Large). For generation, CosyVoice employs a supervised semantic speech tokenizer (S3), an autoregressive language model for token prediction, and a flow-matching-based diffusion model for Mel-spectrogram reconstruction.