Dynamic Mixture-of-Depths for Stacked Channel Bridging
Implement a dynamic mixture-of-depths routing mechanism in the connector module to selectively attend to relevant vision-language layers based on prompt complexity.
Source paper
Dianyi Wang, Ruihang Li, Feng Han, Chaofan Ma, Wei Song, Siyuan Wang, Yibin Wang, Yi Xin, Hongjian Liu, Zhixiong Zhang, Shengyuan Ding, Tianhang Wang, Zhenglin Cheng, Tao Lin, Cheng Jin, Kaicheng Yu, Jingjing Chen, Wenjie Wang, Zhongyu Wei, Jiaqi Wang
Generated by AI. The paper’s authors did not write these.
Implement a dynamic mixture-of-depths routing mechanism in the connector module to selectively attend to relevant vision-language layers based on prompt complexity.
Introduce an auxiliary decoding loss during training to map implicit reasoning think tokens to explicit ground-truth text tokens for improved interpretability.
Expand the reinforcement learning reward system into a mixture of specialized weak supervisor experts dynamically activated by prompt keywords.
The paper introduces DeepGen 1.0, a lightweight (5B parameter) unified multimodal model capable of image generation, editing, and reasoning. By employing a novel Stacked Channel Bridging (SCB) architecture to fuse Vision-Language Model (VLM) features with a Diffusion Transformer (DiT) and utilizing a three-stage training pipeline culminating in Reinforcement Learning (MR-GRPO), the model achieves performance competitive with significantly larger closed-source and open-source models.
DeepGen 1.0 integrates a Qwen-2.5-VL (3B) backbone with an SD3.5-Medium (2B) DiT generator. The core innovation is Stacked Channel Bridging (SCB), which extracts hidden states from six uniformly distributed VLM layers, fuses them with learnable 'think tokens' via channel concatenation, and projects them into the DiT's condition space. Training involves Alignment Pre-training (connector only), Joint Supervised Fine-tuning (VLM LoRA + DiT), and Multi-Reward Group Relative Policy Optimization (MR-GRPO) to align with human preferences.