Skip to content
AI-generated research idea⛔Very hard🔬Significant ResearchRelated work found

Linear-Complexity Backbone via State Space Models

Replace transformer self-attention layers with selective state space model blocks so time-series foundation models scale linearly with patch length during inference.

Time

1-2 weeks

Compute level

Large cluster

Resources

A 16-core TPUv5e cluster for full pre-training on the 300B+ timepoint corpus from scratch.

What the paper used

200M parameters, O(100B) timepoints, TPUv5e setup with 16 tensor-cores

Prerequisites

advanced deep learning architectures (SSMs/Mamba)

Scope

new model architecture, build from scratch (training pipeline)

The Gap

The model uses a 'Stacked Transformer' (Section 4) with multi-head self-attention, which scales quadratically with the number of patches. Appendix A.1 explicitly suggests trying 'efficient linear state space models like Mamba'.

Limitation

While patching reduces sequence length, the quadratic complexity of Attention still limits the ability to process extremely long historical contexts (e.g., high-frequency tick data) efficiently.

Proposed Solution

Replace the Transformer attention layers with Selective State Space Model (SSM) blocks (e.g., Mamba). Retain the patching mechanism and the decoder-only training objective. Pre-train this 'TimesMamba' architecture on the same 100B token corpus to benchmark the trade-off between inference throughput (linear scaling) and zero-shot forecasting accuracy.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

A linear-time foundation model could handle vastly longer context windows, potentially capturing long-term dependencies (e.g., year-over-year seasonality in hourly data) that strictly windowed Transformers miss.

How to Get Started

How to test it: We hypothesize that replacing self-attention with Mamba blocks scales linearly while maintaining zero-shot MAE on ETT datasets within 5 percent of TimesFM. If accuracy degrades beyond 5 percent, we stop.

  1. Replace the stacked transformer attention layers in the base architecture with Mamba state space model blocks.
  2. Retain the original patch embedding mechanism and decoder-only pre-training objective.
  3. Pre-train the new state space architecture on the standard token corpus using a distributed compute cluster.
  4. Benchmark inference throughput and zero-shot forecasting accuracy against the original transformer-based model.

Code linked to this paper on Hugging Face: github.com/google-research/timesfm

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

State space models such as Mamba and MambaTS explore replacing transformer architectures with linear-complexity backbones for time series forecasting. However, none of the listed papers specifically evaluate the exact 100B token pre-training and zero-shot benchmark proposed under the 'TimesMamba' architecture name.

We searched OpenAlex for “Replace transformer self-attention layers with selective state space model blocks so time-series foundation models scale linearly with patch length during inference.” and “Linear-Complexity Backbone via State Space Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

A decoder-only foundation model for time-series forecasting

Abhimanyu Das, Weihao Kong, Rajat Sen +1 more · 2023

This paper introduces TimesFM, a 200M-parameter decoder-only foundation model for time-series forecasting pretrained on a massive corpus of 100 billion real and synthetic time-points. The authors demonstrate that by using a patch-based decoder architecture and extensive pretraining, the model achieves zero-shot performance on unseen datasets that rivals or exceeds state-of-the-art supervised models trained specifically for those tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn