Efficient Input-Dependent Queries for Attention Residuals
A method that introduces low-rank projections and speculative decoding to input-dependent queries in attention residuals, reducing parameter cost and eliminating sequential decoding bottlenecks.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 5.3's component ablation, the authors reveal that projecting the query from the current hidden state (input-dependent query) lowers validation loss to 1.731, outperforming the 1.737 baseline of Full AttnRes. However, they abandoned this because it introduces a d x d projection per layer and forces sequential memory access during autoregressive decoding.
“This further lowers loss to 1.731, but introduces a d × d projection per layer and requires sequential memory access during decoding, so we default to the learned query.”
Limitation
The current pseudo-query is a static, learned parameter for each layer. It cannot dynamically determine what prior layer features to fetch based on the specific token's current representation.
Proposed Solution
Implement a low-rank (LoRA-style) projection for the input-dependent query to reduce the parameter cost. To solve the sequential decoding penalty, utilize speculative decoding or a lookahead projection mechanism where layer l's query is predicted from layer l-k's state, allowing batched query computation while retaining input-dependence.
Why This Matters
Unlocks an additional performance boost by making depth-wise retrieval dynamic to the token context, while preserving the inference speed necessary for production deployment.
How to Get Started
How to test it: Hypothesis: a low-rank query projection predicted from layer l-k enables batched Phase 1 inter-block reads while recovering most of the loss gain of full-rank input-dependent queries (1.731 vs. 1.737/1.746); compare against static Block AttnRes (1.746) and full input-dependent Full AttnRes (1.731) on the 16-layer MoE setup using validation loss, parameter overhead, and decoding latency. Stop if validation loss does not improve over static Block AttnRes (1.746) or if decoding latency overhead exceeds 5% (eroding Block AttnRes's <2% budget).
- Implement a low-rank (LoRA-style) projection mechanism for input-dependent queries within an existing attention residual codebase.
- Establish a baseline evaluating the validation loss and sequential decoding latency of the input-dependent query configuration.
- Integrate a lookahead prediction mechanism to estimate layer l's query from layer l-k's hidden state.
- Benchmark the modified model on autoregressive generation speed to ensure batched query computation improves inference throughput.
Code linked to this paper on Hugging Face: github.com/MoonshotAI/Attention-Residuals
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models and SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation investigate recurrent states and speculative mechanisms in hybrid linear attention models. However, they do not specifically implement a low-rank projection for input-dependent queries paired with a lookahead prediction mechanism to bypass the sequential decoding penalty in attention residuals. They leave the exact query prediction formulation for attention residuals open.
- ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models · 2026 · 1 citations
- Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves · 2026 · 15 citations
- SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation · 2026 · 12 citations
We searched OpenAlex for “A method that introduces low-rank projections and speculative decoding to input-dependent queries in attention residuals, reducing parameter cost and eliminating sequential decoding bottlenecks.” and “Efficient Input-Dependent Queries for Attention Residuals” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Attention ResidualsKimi Team, Guangyu Chen, Yu Zhang +34 more · 2026
This paper introduces Attention Residuals (AttnRes) to address the limitations of standard residual connections, which uniformly aggregate all prior layer outputs and cause uncontrolled hidden-state growth. The authors propose replacing fixed accumulation with learned, depth-wise softmax attention, allowing layers to selectively retrieve information from earlier layers to improve downstream performance.
Related Ideas
- Prompt-Relevant Historical Memory Retrieval and Entropy-Calibrated Layer-Head KV CompressionMovieRoll: A Systematic Framework for Rolling Out Long-Form Videos🟡Fairly easy
- Task-Conditioned Query-Based Decoding for Fine-Grained SemanticsUtonia: Toward One Encoder for All Point Clouds🟠Medium
- Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared AttentionMulti-Branch Transformer🟠Medium
- Controlled Block-Level Comparison of Content-Gated Recurrent Depth ReadersLoopback Transformer: Towards a Unified View of Residual Looped Transformers🟡Fairly easy
- Granular Importance-Aware Sparse RetrievalLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference🔴Hard