Linear-Complexity Depth-Wise Attention for Scalable Full Attention Residuals
A method that replaces depth-wise softmax attention with linear-complexity kernelized attention along the depth dimension, allowing networks to scale residual connections efficiently without quadratic memory overhead.