Skip to content
AI-generated research idea🔴Hard🔬Significant Research

Scaling Spatial Test-Time Training to In-the-Wild Videos via Self-Supervised Depth Estimation

Replace explicit 3D scene-description supervision with self-supervised monocular depth pseudo-labels to provide dense gradient signals for spatial test-time training fast weights.

A very similar idea came first from another paper: Self-Supervised Geometric Distillation for Long-Horizon Data Walls

Time

3-4 weeks (1 week for video curation and offline depth pseudo-label generation, 1.5-2 weeks for Stage-1/Stage-2 training and loss-weighting ablations, and 3-5 days for benchmark evaluation including outdoor test splits).

Compute level

Large cluster

Resources

8x 80GB A100/H100 GPUs for running Depth Anything pseudo-labeling across ~16k-50k unconstrained video clips, Stage-1 dense depth training (32 frames), and Stage-2 3M-sample spatial VQA fine-tuning (64-128 frames), along with ~4 TB storage for raw video clips and cached depth maps.

What the paper used

2B model initialized from Qwen3-VL-2B-Instruct; Stage 1 trains on ~16k dense scene-description samples (3.6k ScanNet, 12.5k ARKitScenes) at 32 frames, and Stage 2 fine-tunes on ~3M spatial VQA samples (~2.5M open-sourced and ~0.5M self-collected, also noted as 2M in Sec. 3.4) at 64-128 frames; evaluation and memory benchmarks run on an 80GB GPU (peak 11.9 GB and 799.4 TFLOPs at 1024 frames); training GPU count and hours are not stated in the paper.

Prerequisites

computer vision

Scope

large-scale data collection

The Gap

The model relies heavily on a custom dense scene-description dataset derived from explicit object-centric 3D scene graphs (ScanNet, ARKitScenes) to provide rich, high-coverage gradient signals for learning the fast-weight dynamics during the first training stage.

“Unlike short-form answers, the target output is formatted as a coherent scene walkthrough, which is derived from object-centric 3D scene graphs (Jia et al., 2024).”
From the paper, Section 3.3, page 6. Matched against the PDF text.

Limitation

The requirement for detailed 3D scene graphs and dense walkthroughs restricts the training data to indoor, pre-mapped environments, limiting the model's generalizability and ability to scale to unconstrained, in-the-wild outdoor videos.

Proposed Solution

Replace or augment the explicit 3D scene-description supervision with a self-supervised auxiliary task, such as monocular depth estimation using pseudo-labels from models like Depth Anything, to provide dense gradient signals for the TTT fast weights without requiring manual 3D annotations.

Why This Matters

Removing the bottleneck of expensive, domain-specific 3D annotations allows the spatial memory capabilities to scale to massive, diverse video datasets, dramatically improving real-world generalization.

How to Get Started

How to test it: Hypothesis: Stage-1 supervision using monocular depth pseudo-labels (e.g., from Depth Anything) on unconstrained videos teaches fast-weight update dynamics without 3D scene-graph walkthroughs. Compare depth-supervised Stage-1 + Stage-2 Spatial-TTT-2B against the original Spatial-TTT-2B (64.4 Avg.) and w/o Dense Data ablation (61.3 Avg.) on VSI-Bench and an outdoor/in-the-wild spatial video split; stop if it fails to beat the w/o Dense Data baseline (61.3 Avg.) on VSI-Bench or shows no gain on outdoor spatial reasoning.

  1. Generate pseudo-labels for in-the-wild video datasets using off-the-shelf monocular depth estimators like Depth Anything.
  2. Adapt the Spatial-TTT training objective to accept self-supervised depth pseudo-labels in place of 3D scene graph annotations.
  3. Train the fast-weight dynamics on unannotated video sequences using the modified self-supervised loss function.
  4. Evaluate generalization performance on diverse in-the-wild video benchmarks compared to models trained exclusively on ScanNet or ARKitScenes.

Code linked to this paper on Hugging Face: github.com/THU-SI/Spatial-TTT

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Not checked against the literature yet.

Source Paper

Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training

Fangfu Liu, Diankun Wu, Jiawei Chi +7 more · 2026

Spatial-TTT introduces a novel framework for streaming visual-based spatial intelligence in Multimodal Large Language Models (MLLMs) by utilizing Test-Time Training (TTT). The authors' main contribution is enabling MLLMs to selectively maintain, progressively update, and reason over 3D spatial evidence from potentially unbounded long-horizon video streams.

Related Ideas

Is this a good idea?

Share onXLinkedIn