Closed-Loop Synchronization for Long-Horizon Consistency
Establish a closed-loop synchronization feedback mechanism where estimated depth and pose from generated video frames actively correct the underlying physics simulator state.
Source paper
Wei Liu, Ziyu Chen, Zizhang Li, Yue Wang, Hong-Xing Yu, Jiajun Wu
Generated by AI. The paper’s authors did not write these.
Establish a closed-loop synchronization feedback mechanism where estimated depth and pose from generated video frames actively correct the underlying physics simulator state.
Replace monocular depth map reconstruction with a large reconstruction model outputting 3D Gaussian Splats, providing accurate geometric initialization for physics simulations.
Introduce a physics-consistency loss during distillation that penalizes generated video frames when optical flow deviates from projected particle trajectories in the simulator.
RealWonder proposes a real-time system for generating videos conditioned on 3D physical actions (forces, gravity, collisions) from a single image. The method bridges the gap between abstract physical actions and visual generation by using a physics simulator to produce intermediate optical flow and coarse RGB previews, which then condition a distilled, fast-inference video diffusion model.
The pipeline consists of three stages: (1) Single-image 3D scene reconstruction using depth estimation and point clouds; (2) Real-time physics simulation (using MPM/PBD solvers) to predict dynamics and render optical flow/coarse previews; and (3) A flow-conditioned video generation model, distilled from a larger diffusion model into a 4-step causal generator for real-time streaming.