Non-Autoregressive Local Decoding for Real-Time Inference
Replace the autoregressive local transformer with a non-autoregressive module to predict residual codebook tokens in parallel and reduce real-time inference latency.
Source paper
Dongchao Yang, Yuxin Xie, Yuguo Yin, Zheyu Wang, Xiaoyu Yi, Gongxi Zhu, Xiaolong Weng, Zihan Xiong, Yingzhe Ma, Dading Cong, Jingliang Liu, Zihang Huang, Jinghan Ru, Rongjie Huang, Haoran Wan, Peixu Wang, Kuoxi Yu, Helin Wang, Liming Liang, Xianwei Zhuang, Yuanyuan Wang, Haohan Guo, Junjie Cao, Zeqian Ju, Songxiang Liu, Yuewen Cao, Heming Weng, Yuexian Zou
Generated by AI. The paper’s authors did not write these.
Replace the autoregressive local transformer with a non-autoregressive module to predict residual codebook tokens in parallel and reduce real-time inference latency.
Incorporate lightweight speaker identity adapters into cross-attention layers to enable authorized zero-shot voice cloning without retraining the base model.
Integrate a ControlNet-style side network to accept dense time-series control inputs for frame-level guidance during audio generation.
Implement a variable frame-rate tokenizer to capture high-speed phonetic details in complex lyrics while keeping background audio compressed at a lower rate.
HeartMuLa introduces a comprehensive open-source ecosystem for music generation, featuring a low-frame-rate audio tokenizer (HeartCodec), a lyric aligner, and a 3B parameter autoregressive generation model. By factorizing generation into global structural modeling and local acoustic refinement, and employing a four-stage training pipeline including Direct Preference Optimization (DPO), the system achieves state-of-the-art long-form music synthesis (up to 6 minutes) with high lyrical intelligibility and style control.
The model uses a hierarchical autoregressive architecture based on the Llama-2 backbone. It operates on 12.5 Hz discrete audio tokens extracted by HeartCodec, which combines semantic features from Whisper/WavLM with a Flow Matching decoder. Generation is split into a Global Transformer (predicting coarse semantic tokens) and a Local Transformer (refining acoustic details), conditioned on lyrics, style tags, and reference audio.