Improving Temporal Consistency in World Models via Physics-Informed Dynamic Masking
A method that applies a physics-based 3D tracking pre-filtering step to mask out temporally inconsistent dynamic objects, improving world model consistency across irregular capture intervals.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The training data pipeline synthesizes temporally continuous video from sparse street-view images captured at equal distance intervals rather than equal time intervals. When the capture vehicle slows down or stops, consecutive frames span large temporal gaps.
“When these sequences are converted into interpolated training video, dynamic objects such as vehicles may abruptly appear or disappear between frames.”
Limitation
Because of these varying temporal gaps and noisy capture metadata, the interpolated training videos occasionally contain temporally inconsistent dynamic objects, causing vehicles and pedestrians to abruptly appear or disappear. The model absorbs these artifacts during training, leading to sudden object popping in inference.
Proposed Solution
Implement an optical-flow or physics-based 3D tracking pre-filtering step to detect and mask out temporally inconsistent dynamic objects (e.g., cars moving unnaturally between large time gaps) prior to view interpolation. Alternatively, replace the intermittent freeze-frame technique with a 4D feature-splatting interpolation network that enforces object permanence across large temporal gaps.
Why This Matters
Eliminating sudden object disappearances and appearances will significantly increase the physical realism and temporal coherence of the generated simulation, making it viable for generating highly reliable autonomous driving training data.
How to Get Started
How to test it: Hypothesis: Pre-filtering/masking inconsistent dynamic objects with optical-flow/3D tracking (or using 4D feature-splatting) before view interpolation eliminates object popping in SWM inference without hurting static geometry grounding. Compare against SWM (TF) trained on the standard intermittent freeze-frame pipeline using FID, FVD, mPSNR/mLPIPS on Busan-City-Bench and Ann-Arbor-City-Bench plus a dynamic-object popping rate; stop if FVD worsens by >5% or object-popping frequency drops by <20% relative to baseline.
- Analyze the temporal gaps between consecutive frames in street-view driving datasets like nuScenes or Waymo.
- Implement an optical-flow or physics-based 3D tracking algorithm to detect large time gaps and moving objects.
- Apply dynamic masking to filter out temporally inconsistent frames prior to model training.
- Train a baseline world model on the filtered dataset and measure improvements in video consistency and object permanence.
Code linked to this paper on Hugging Face: github.com/naver-ai/seoul-world-model
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like "Enhancing Physical Consistency in Lightweight World Models" and "DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Masking" explore physics-informed approaches and spatial-temporal masking for consistency, but they do not specifically implement a 3D tracking pre-filtering or 4D feature-splatting interpolation network for world models as proposed.
- Enhancing Physical Consistency in Lightweight World Models · 2025 · 9 citations
- DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Masking · 2026 · 1 citations
We searched OpenAlex and arXiv for “A method that applies a physics-based 3D tracking pre-filtering step to mask out temporally inconsistent dynamic objects, improving world model consistency across irregular capture intervals.” and “Improving Temporal Consistency in World Models via Physics-Informed Dynamic Masking” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Grounding World Simulation Models in a Real-World MetropolisJunyoung Seo, Hyunwook Choi, Minkyung Kwon +10 more · 2026
The paper introduces the Seoul World Model (SWM), a city-scale video generative world model that anchors simulated environments to the actual physical layout of a real metropolis. The main contribution is a retrieval-augmented generation framework that leverages cross-temporal pairing, synthetic urban data, and a Virtual Lookahead Sink to synthesize spatially faithful, temporally consistent, and long-horizon videos.
Related Ideas
- 4D Spatiotemporal Pretraining for Dynamic ScenesUtonia: Toward One Encoder for All Point Clouds🔴Hard
- Natural Predictive Pre-training on Web-Scale VideoIn Pursuit of Pixel Supervision for Visual Pre-training⛔Very hard
- Unsupervised Motion Disambiguation via Semantic Scene Priorsπ^4: A Foundation Model for 4D Scene Geometry and Motion🟠Medium
- Extending Spatial Curation to Dynamic Scenes via 4D RepresentationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🔴Hard
- Dynamic Object Handling via Motion SegmentationWorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion🟡Fairly easy