Covariate-Aware Foundation Model Architecture
Extend univariate time-series foundation models to accept parallel streams of dynamic covariates and static attributes through fused patch embeddings.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The current TimesFM model is strictly univariate (f: y1:L → ŷL+1:L+H), as noted in Appendix A.1. It processes the target time-series history but ignores dynamic covariates (e.g., price, weather) and static attributes (e.g., product category), which are critical in domains like retail and energy.
“Currently the model is not pretrained with covariates as one of the key challenges is finding large volumes of pretrained data with meaningful covariates”
Limitation
The inability to ingest exogenous variables limits the model's applicability in causal forecasting scenarios where the target variable is driven by external factors rather than just its own history.
Proposed Solution
Extend the architecture to accept a parallel stream of covariate patches. Modify the `InputResidualBlock` to fuse the time-series patch embedding with covariate embeddings (via concatenation or cross-attention). Fine-tune the pre-trained model on multivariate datasets (like M5 or Traffic) where covariates are available, or generate synthetic covariates (e.g., random 'price' shocks correlated with the synthetic 'demand') to continue pre-training.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Enabling covariate support would transition the model from a pure extrapolator to a causal forecasting tool, significantly increasing its utility in industrial applications like supply chain planning.
How to Get Started
How to test it: We hypothesize that adding covariate patches improves MAE on multivariate datasets like M5 compared to univariate baselines. If the MAE does not decrease by at least 2 percent after fine-tuning, we stop.
- Modify the input residual block of the base univariate architecture to accept covariate patch embeddings alongside target history.
- Implement cross-attention or concatenation mechanisms to fuse time-series features with external price and weather variables.
- Curate multivariate benchmark datasets such as M5 or Traffic with well-defined dynamic and static covariates.
- Fine-tune the expanded model architecture on the multivariate datasets and evaluate forecasting accuracy against univariate baselines.
Code linked to this paper on Hugging Face: github.com/google-research/timesfm
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like CoRA: Covariate-Aware Adaptation of Time Series Foundation Models and UniCA: Unified Covariate Adaptation for Time Series Foundation Model specifically investigate adapting time-series foundation models to incorporate covariates. These papers explore architectures and adaptation strategies for fusing covariates with pre-trained embeddings for multivariate forecasting. Therefore, the proposed architectural modifications and fine-tuning strategies have essentially been explored in the recent literature.
- CoRA: Covariate-Aware Adaptation of Time Series Foundation Models · 2025 · 12 citations
- UniCA: Unified Covariate Adaptation for Time Series Foundation Model · 2025 · 6 citations
We searched OpenAlex and arXiv for “Extend univariate time-series foundation models to accept parallel streams of dynamic covariates and static attributes through fused patch embeddings.” and “Covariate-Aware Foundation Model Architecture” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
A decoder-only foundation model for time-series forecastingAbhimanyu Das, Weihao Kong, Rajat Sen +1 more · 2023
This paper introduces TimesFM, a 200M-parameter decoder-only foundation model for time-series forecasting pretrained on a massive corpus of 100 billion real and synthetic time-points. The authors demonstrate that by using a patch-based decoder architecture and extensive pretraining, the model achieves zero-shot performance on unseen datasets that rivals or exceeds state-of-the-art supervised models trained specifically for those tasks.
Related Ideas
- Learnable Residual Correction for Latent PredictionFlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Prediction🟡Fairly easy
- Non-Autoregressive Local Decoding for Real-Time InferenceHeartMuLa: A Family of Open Sourced Music Foundation Models🟠Medium
- Bootstrapping RLM-Specific ReasonersRecursive Language Models🟠Medium
- Look-Ahead Causal Attention for Enhanced Cross-Modal SynchronyOmniForcing: Unleashing Real-time Joint Audio-Visual Generation🟠Medium
- Synthetic Trajectory Distillation for Specialized Open-Weight ModelsDeepCode: Open Agentic Coding🟠Medium