Skip to content

Source paper

π^4: A Foundation Model for 4D Scene Geometry and Motion

Yifan Wang, Weiguang Zhao, Ni Yang and 4 more

Yifan Wang, Weiguang Zhao, Ni Yang, Haoyu Guo, Junting Dong, Weinan Zhang, Chunhua Shen

arXiv 2611.23144 · 202611 upvotes on Hugging Face

Research ideas from this paper (2)

Generated by AI. The paper’s authors did not write these.

AI summary

The paper introduces pi4, a unified foundation model for 4D scene perception that reconstructs dense geometry, camera poses, and tracks from uncalibrated RGB observations. By employing a shared DINOv3 encoder, looped Transformer decoders, and a geometry-conditioned motion branch, the model jointly achieves state-of-the-art multi-view scene reconstruction and 3D point tracking.

How it works (AI summary)

The model uses a shared DINOv3 image encoder feeding two independent task-specific looped Transformer decoders for geometry and motion. The geometry branch predicts dense local pointmaps, camera poses, and scale, which are then refined via a sparse 3D U-Net. The motion branch conditions on detached geometry features using spatial AdaLN to predict image-space correspondences that are analytically lifted into 3D using the reconstructed target geometry.