Skip to content

Source paper

Attention Residuals

Kimi Team, Guangyu Chen, Yu Zhang and 34 more

Kimi Team, Guangyu Chen, Yu Zhang, Jianlin Su, Weixin Xu, Siyuan Pan, Yaoyu Wang, Yucheng Wang, Guanduo Chen, Bohong Yin, Yutian Chen, Junjie Yan, Ming Wei, Y. Zhang, Fanqing Meng, Chao Hong, Xiaotong Xie, Shaowei Liu, Enzhe Lu, Yunpeng Tai, Yanru Chen, Xin Men, Haiqing Guo, Y. Charles, Haoyu Lu, Lin Sui, Jinguo Zhu, Zaida Zhou, Weiran He, Weixiao Huang, Xinran Xu, Yuzhi Wang, Guokun Lai, Yulun Du, Yuxin Wu, Zhilin Yang, Xinyu Zhou

arXiv 2603.15031 · 202668 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces Attention Residuals (AttnRes) to address the limitations of standard residual connections, which uniformly aggregate all prior layer outputs and cause uncontrolled hidden-state growth. The authors propose replacing fixed accumulation with learned, depth-wise softmax attention, allowing layers to selectively retrieve information from earlier layers to improve downstream performance.

How it works (AI summary)

AttnRes replaces standard residual additions with a softmax attention mechanism over previous layers using learned pseudo-queries. To scale this efficiently for training and inference, the authors introduce Block AttnRes, which groups layers into blocks to compress the memory and communication overhead from O(Ld) to O(Nd), paired with cross-stage caching and a two-phase inference strategy.