Skip to content

Source paper

HeartMuLa: A Family of Open Sourced Music Foundation Models

Dongchao Yang, Yuxin Xie, Yuguo Yin and 25 more

Dongchao Yang, Yuxin Xie, Yuguo Yin, Zheyu Wang, Xiaoyu Yi, Gongxi Zhu, Xiaolong Weng, Zihan Xiong, Yingzhe Ma, Dading Cong, Jingliang Liu, Zihang Huang, Jinghan Ru, Rongjie Huang, Haoran Wan, Peixu Wang, Kuoxi Yu, Helin Wang, Liming Liang, Xianwei Zhuang, Yuanyuan Wang, Haohan Guo, Junjie Cao, Zeqian Ju, Songxiang Liu, Yuewen Cao, Heming Weng, Yuexian Zou

arXiv 2601.10547 · 202642 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

AI summary

HeartMuLa introduces a comprehensive open-source ecosystem for music generation, featuring a low-frame-rate audio tokenizer (HeartCodec), a lyric aligner, and a 3B parameter autoregressive generation model. By factorizing generation into global structural modeling and local acoustic refinement, and employing a four-stage training pipeline including Direct Preference Optimization (DPO), the system achieves state-of-the-art long-form music synthesis (up to 6 minutes) with high lyrical intelligibility and style control.

How it works (AI summary)

The model uses a hierarchical autoregressive architecture based on the Llama-2 backbone. It operates on 12.5 Hz discrete audio tokens extracted by HeartCodec, which combines semantic features from Whisper/WavLM with a Flow Matching decoder. Generation is split into a Global Transformer (predicting coarse semantic tokens) and a Local Transformer (refining acoustic details), conditioned on lyrics, style tags, and reference audio.