Sparse Scaling via Multimodal Mixture-of-Experts
A method that replaces dense feed-forward networks with sparse mixture-of-experts layers to scale multimodal model capacity without increasing inference compute costs.
Source paper
Huichao Zhang, Liao Qu, Yiheng Liu, Hang Chen, Yangyang Song, Yongsheng Dong, Shikun Sun, Xian Li, Xu Wang, Yi Jiang, Hu Ye, Bo Chen, Yiming Gao, Peng Liu, Akide Liu, Zhipeng Yang, Qili Deng, Linjie Xing, Jiyang Liu, Zhao Wang, Yang Zhou, Mingcong Liu, Yi Zhang, Qian He, Xiwei Hu, Zhongqi Qi, Jie Shao, Zhiye Fu, Shuai Wang, Fangmin Chen, Xuezhi Chai, Zhihua Wu, Yitong Wang, Zehuan Yuan, Daniel K. Du, Xinglong Wu
Generated by AI. The paper’s authors did not write these.
A method that replaces dense feed-forward networks with sparse mixture-of-experts layers to scale multimodal model capacity without increasing inference compute costs.
A system that allocates variable token densities to images using dynamic compression to eliminate fixed-grid resolution bottlenecks in visual tokenizers.
A method that uses a phased curriculum and fidelity reward to stabilize the reinforcement learning of fine-grained scales in generative models.
This paper introduces NextFlow, a unified 7B-parameter decoder-only transformer capable of both multimodal understanding and generation. By utilizing a "next-scale" prediction paradigm (coarse-to-fine) rather than traditional raster scanning, and a dual-codebook tokenizer, the model achieves state-of-the-art generation speed and visual quality while maintaining the reasoning capabilities of Large Language Models.
The model uses a decoder-only transformer architecture initialized from Qwen2.5-VL. For visual generation, it employs a Vector Quantized (VQ) approach with a "next-scale" prediction strategy, generating images layer-by-layer (from coarse layout to fine detail) which allows for O(log N) inference steps. It uses a dual-codebook tokenizer to align semantic and pixel-level features and incorporates a specific Reinforcement Learning strategy (GRPO with prefix-tuning) to align the model's visual outputs with human preferences.