Hierarchical Decay Mechanisms for Infinite-Context Memory
Implement multi-scale fast weights with varying decay rates to maintain both local detail and global context in infinite-sequence memory models.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 4.6 and Supplementary Material A, the authors note that the 'fast-weight memory has a fixed size,' which leads to forgetting earlier inputs in long sequences. Empirical analysis showed performance degradation in high-complexity scenes because the single set of fast weights cannot retain infinite history.
“Our fast-weight memory has a fixed size, which may limit its ability to handle highly complex scenarios with extremely large numbers of input views.”
Limitation
The current architecture uses a single update rule for fast weights, causing a 'forgetting' effect where recent observations overwrite earlier geometric structure in long autoregressive streams.
Proposed Solution
Implement a multi-scale TTT layer structure where different 'heads' of the fast weights utilize different decay rates (e.g., exponential moving averages with varying betas). One set of weights updates rapidly for local details, while another updates slowly to preserve global scene geometry. This mimics 'short-term' and 'long-term' memory.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Crucial for scaling the model to city-scale or 'infinite' streaming reconstruction (e.g., autonomous driving), ensuring the map does not degrade over time.
How to Get Started
How to test it: We hypothesize that a multi-scale TTT layer with varying decay rates will improve long-context memory retention. We will compare against the single update rule baseline using PSNR and SSIM on DL3DV-140 scenes, stopping if the multi-scale decay fails to yield at least a 0.5 dB PSNR improvement.
- Review the existing test-time training codebase and identify the single fast-weight update equations.
- Define multiple parallel fast-weight heads configured with distinct exponential moving average decay rates.
- Modify the layer forward pass to combine fast updates for local geometry and slow updates for global scene structure.
- Evaluate long-sequence reconstruction fidelity and task performance against single-scale baselines.
Code linked to this paper on Hugging Face: github.com/cwchenwang/tttLRM
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers implement a multi-scale Test-Time Training (TTT) layer structure featuring heads with distinct decay rates for local and global context. While works like Memformer and ∞-former address memory-augmented transformers and infinite context, and Zebra uses grouped local-global attention, none explore hierarchical decay mechanisms within fast weight updates.
- Memformer: A Memory-Augmented Transformer for Sequence Modeling · 2022 · 111 citations
- ∞-former: Infinite Memory Transformer-former: Infinite Memory Transformer · 2022 · 49 citations
- Landmark Attention: Random-Access Infinite Context Length for Transformers · 2023 · 181 citations
We searched OpenAlex for “Implement multi-scale fast weights with varying decay rates to maintain both local detail and global context in infinite-sequence memory models.” and “Hierarchical Decay Mechanisms for Infinite-Context Memory” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
tttLRM: Test-Time Training for Long Context and Autoregressive 3D ReconstructionChen Wang, Hao Tan, Wang Yifan +6 more · 2026
The paper presents tttLRM, a Large Reconstruction Model that utilizes Test-Time Training (TTT) layers to perform 3D reconstruction from long sequences of images with linear computational complexity. It enables both feedforward and autoregressive (streaming) reconstruction of explicit 3D representations like Gaussian Splats, scaling efficiently to large context windows where traditional Transformers fail.
Related Ideas
- Infinite-Context Linear State Generalization for Feedforward ReconstructionLoGeR: Long-Context Geometric Reconstruction with Hybrid Memory🟠Medium
- Infinite-Horizon Attention via Compressed Memory Caching in Hierarchical DenoisingHiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising🔴Hard
- Recurrent Visual Memory for Infinite-Context TrajectoriesMobile-Agent-v3: Foundamental Agents for GUI Automation🔴Hard
- Adaptive Attention-Based Memory CompressionHelios: Real Real-Time Long Video Generation Model🟠Medium
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium