Skip to content

Source paper

Multi-Branch Transformer

Gaotang Li, Lingjie Chen, Chunyuan Deng and 6 more

Gaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang, Hanghang Tong

arXiv 2611.19800 · 202643 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces the Multi-Branch Transformer (MBT), an architectural paradigm that parallelizes the sequential execution of Transformer blocks by routing representations from shared prelude layers through multiple independently parameterized parallel branches before fusing them in shared coda layers. To optimize inference at long contexts, the authors introduce Joint-KV, which computes a unified key-value cache across concurrent branches per depth level, alongside branch grouping and query head folding within an optimized inference runtime. Experiments on 0.8B to 2.1B parameter MoE models show that MBTs consistently improve language modeling loss and zero-shot benchmark performance under matched compute, while reducing KV cache storage up to 34% and increasing decode throughput on an H200 GPU by up to 51.4%.

How it works (AI summary)

MBT restructures the middle layers of a Transformer into K parallel, concurrently executed block branches between shared prelude and coda layers, fusing outputs using RMSNorm concatenation followed by a linear projection. Under the Balanced allocation strategy, branch width and sequential depth are co-optimized under matched training FLOPs. Furthermore, Joint-KV shares key-value states across all K branches at each layer depth by projecting concatenated normalized branch representations into a single KV cache bank, which is paired with custom inference serving techniques (branch grouping to batch operator kernels and head folding to reuse KV loads across branches in a single grouped-query attention kernel).