Skip to content
AI-generated research idea🔴Hard💡Moderate InnovationRelated work found

Improving Temporal Consistency in World Models via Physics-Informed Dynamic Masking

A method that applies a physics-based 3D tracking pre-filtering step to mask out temporally inconsistent dynamic objects, improving world model consistency across irregular capture intervals.

Time

Approximately 1 to 2 weeks (3-5 days for dynamic-object filtering and video re-interpolation over the 440K-image dataset, 2-4 days for 10K DiT training iterations on 24 H100 GPUs, and 1-2 days for benchmark evaluation).

Compute level

Large cluster

Resources

24 NVIDIA H100 GPUs for retraining the 2B-parameter DiT (10K iterations, batch size 48) on 440K street-view images, 12.7K synthetic videos, and Waymo data, plus multi-GPU compute to run 3D tracking/optical-flow filtering and re-run view interpolation across the street-view sequences.

What the paper used

Fine-tunes Cosmos-Predict2.5-2B (2B-parameter DiT) using AdamW for 10K iterations at batch size 48 across 24 NVIDIA H100 GPUs (plus 6K ODE init steps on 1K pairs and 10K fine-tuning iterations for the Self-Forcing variant; inference runs at 15.2 fps on 1 H100 GPU) on 440K Seoul street-view images (from 1.2M panoramas), 12.7K CARLA synthetic videos, Waymo Open Dataset, and 10K stylized videos.

Prerequisites

computer vision

Scope

large-scale data collection

The Gap

The training data pipeline synthesizes temporally continuous video from sparse street-view images captured at equal distance intervals rather than equal time intervals. When the capture vehicle slows down or stops, consecutive frames span large temporal gaps.

“When these sequences are converted into interpolated training video, dynamic objects such as vehicles may abruptly appear or disappear between frames.”
From the paper, Appendix D.1, page 29. Matched against the PDF text.

Limitation

Because of these varying temporal gaps and noisy capture metadata, the interpolated training videos occasionally contain temporally inconsistent dynamic objects, causing vehicles and pedestrians to abruptly appear or disappear. The model absorbs these artifacts during training, leading to sudden object popping in inference.

Proposed Solution

Implement an optical-flow or physics-based 3D tracking pre-filtering step to detect and mask out temporally inconsistent dynamic objects (e.g., cars moving unnaturally between large time gaps) prior to view interpolation. Alternatively, replace the intermittent freeze-frame technique with a 4D feature-splatting interpolation network that enforces object permanence across large temporal gaps.

Why This Matters

Eliminating sudden object disappearances and appearances will significantly increase the physical realism and temporal coherence of the generated simulation, making it viable for generating highly reliable autonomous driving training data.

How to Get Started

How to test it: Hypothesis: Pre-filtering/masking inconsistent dynamic objects with optical-flow/3D tracking (or using 4D feature-splatting) before view interpolation eliminates object popping in SWM inference without hurting static geometry grounding. Compare against SWM (TF) trained on the standard intermittent freeze-frame pipeline using FID, FVD, mPSNR/mLPIPS on Busan-City-Bench and Ann-Arbor-City-Bench plus a dynamic-object popping rate; stop if FVD worsens by >5% or object-popping frequency drops by <20% relative to baseline.

  1. Analyze the temporal gaps between consecutive frames in street-view driving datasets like nuScenes or Waymo.
  2. Implement an optical-flow or physics-based 3D tracking algorithm to detect large time gaps and moving objects.
  3. Apply dynamic masking to filter out temporally inconsistent frames prior to model training.
  4. Train a baseline world model on the filtered dataset and measure improvements in video consistency and object permanence.

Code linked to this paper on Hugging Face: github.com/naver-ai/seoul-world-model

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like "Enhancing Physical Consistency in Lightweight World Models" and "DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Masking" explore physics-informed approaches and spatial-temporal masking for consistency, but they do not specifically implement a 3D tracking pre-filtering or 4D feature-splatting interpolation network for world models as proposed.

We searched OpenAlex and arXiv for “A method that applies a physics-based 3D tracking pre-filtering step to mask out temporally inconsistent dynamic objects, improving world model consistency across irregular capture intervals.” and “Improving Temporal Consistency in World Models via Physics-Informed Dynamic Masking” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Grounding World Simulation Models in a Real-World Metropolis

Junyoung Seo, Hyunwook Choi, Minkyung Kwon +10 more · 2026

The paper introduces the Seoul World Model (SWM), a city-scale video generative world model that anchors simulated environments to the actual physical layout of a real metropolis. The main contribution is a retrieval-augmented generation framework that leverages cross-temporal pairing, synthetic urban data, and a Virtual Lookahead Sink to synthesize spatially faithful, temporally consistent, and long-horizon videos.

Related Ideas

Is this a good idea?

Share onXLinkedIn