Mixture-of-Experts (MoE) DiT for Latency-Constrained Scaling
A system that replaces dense feed-forward networks in diffusion backbones with Mixture-of-Experts layers to scale parameters while maintaining real-time latency.
Source paper
Xiaofeng Mao, Zhen Li, Chuanhao Li, Xiaojie Xu, Kaining Ying, Tong He, Jiangmiao Pang, Yu Qiao, Kaipeng Zhang
Generated by AI. The paper’s authors did not write these.
A system that replaces dense feed-forward networks in diffusion backbones with Mixture-of-Experts layers to scale parameters while maintaining real-time latency.
A method that integrates a frozen optical flow network during fine-tuning to penalize motion directions that oppose user control inputs.
A method that uses lightweight saliency maps to dynamically allocate history tokens to high-information regions while compressing static backgrounds.
This paper presents Yume1.5, a multimodal framework designed to generate infinite, interactive, and photorealistic video worlds from a single image or text prompt. The authors address limitations in latency and context length by introducing a Joint Temporal-Spatial-Channel Modeling (TSCM) strategy for efficient history compression and a Self-Forcing training paradigm to mitigate autoregressive error accumulation.
The model utilizes a Video Diffusion Transformer (based on Wan 2.1) modified with TSCM, which splits historical context compression into two pathways: temporal-spatial compression (via random frame sampling and patchify) and channel-wise compression (via Linear Attention). It employs a distillation-based acceleration method combining score distillation and a 'Self-Forcing' technique where the model trains on its own predictions to enable robust 4-step real-time inference.