Skip to content

Source paper

Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training

Fangfu Liu, Diankun Wu, Jiawei Chi and 7 more

Fangfu Liu, Diankun Wu, Jiawei Chi, Yimo Cai, Yi-Hsin Hung, Xumin Yu, Hao Li, Han Hu, Yongming Rao, Yueqi Duan

arXiv 2603.12255 · 202666 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

Spatial-TTT introduces a novel framework for streaming visual-based spatial intelligence in Multimodal Large Language Models (MLLMs) by utilizing Test-Time Training (TTT). The authors' main contribution is enabling MLLMs to selectively maintain, progressively update, and reason over 3D spatial evidence from potentially unbounded long-horizon video streams.

How it works (AI summary)

The framework employs a hybrid architecture that interleaves TTT layers with standard self-attention anchor layers at a 3:1 ratio to preserve pretrained knowledge. It utilizes large-chunk updates and sliding-window attention for parallel efficiency, incorporates a spatial-predictive 3D spatiotemporal convolution to capture geometric structure, and is supervised using a custom dense 3D scene-description dataset.