Adaptive Reference-Free Extrapolation for Video World Models under Sparse Grounding
Build an adaptive conditioning mechanism that dynamically shifts attention between autoregressive history and text prompts when spatial reference density drops below a threshold.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The geometric and semantic referencing mechanisms rely heavily on retrieving an optimal number of nearest street-view references (K=5) per chunk. However, real-world reference density is often non-uniform.
“As K decreases, mPSNR drops, since fewer references provide less coverage of the target scene.”
Limitation
When the number of retrieved references (K) drops due to sparse mapping, the spatial adherence metric (mPSNR) degrades significantly because the model lacks sufficient visual coverage of the target scene, forcing suboptimal hallucination that misaligns with physical reality.
Proposed Solution
Develop an adaptive, confidence-based conditioning mechanism. When reference density or quality drops below a threshold, the model should dynamically scale up the attention weight of the autoregressive context (history latents) and text-prompt guidance. Train a transition layer that seamlessly switches between retrieval-augmented spatial grounding and purely generative extrapolation without visual seams.
Why This Matters
This enhancement allows the world model to operate continuously and smoothly even when navigating outside of heavily mapped urban centers into unmapped or sparsely documented rural environments.
How to Get Started
How to test it: Hypothesis: Dynamically scaling attention to autoregressive history latents and text prompts via a confidence-gated transition layer when K is low improves visual continuity and reduces seams compared to static reference dropout. Compare against standard SWM (which uses 20% reference dropout) at K=0, 1, 2 and on dense-to-sparse transition trajectories on Busan-City-Bench using sliding-window FID/FVD, VBench Image Quality, RotErr/TransErr, and mPSNR; stop if gating does not outperform the baseline on FVD or dense-to-sparse seam smoothness at K<=1.
- Audit the existing video generation codebase to locate where reference retrieval indices are computed per chunk.
- Implement a threshold-based variance and density metric to measure local reference quality.
- Write a transition blending module to smoothly interpolate spatial grounding weights with historical context latents.
- Fine-tune the attention gating layer on a subset of the Waymo ornuScenes datasets featuring sparse annotation regions.
Code linked to this paper on Hugging Face: github.com/naver-ai/seoul-world-model
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers address an adaptive, confidence-based conditioning mechanism for dynamically scaling autoregressive context and text prompts under sparse grounding during video extrapolation.
We searched OpenAlex for “Build an adaptive conditioning mechanism that dynamically shifts attention between autoregressive history and text prompts when spatial reference density drops below a threshold.” and “Adaptive Reference-Free Extrapolation for Video World Models under Sparse Grounding” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Grounding World Simulation Models in a Real-World MetropolisJunyoung Seo, Hyunwook Choi, Minkyung Kwon +10 more · 2026
The paper introduces the Seoul World Model (SWM), a city-scale video generative world model that anchors simulated environments to the actual physical layout of a real metropolis. The main contribution is a retrieval-augmented generation framework that leverages cross-temporal pairing, synthetic urban data, and a Virtual Lookahead Sink to synthesize spatially faithful, temporally consistent, and long-horizon videos.
Related Ideas
- Unsupervised Motion Disambiguation via Semantic Scene Priorsπ^4: A Foundation Model for 4D Scene Geometry and Motion🟠Medium
- Self-Supervised Geometric Distillation for Long-Horizon Data WallsLoGeR: Long-Context Geometric Reconstruction with Hybrid Memory🔴Hard
- Global Geometric Memory for Loop ClosureWorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion🟠Medium
- Natural Predictive Pre-training on Web-Scale VideoIn Pursuit of Pixel Supervision for Visual Pre-training⛔Very hard
- Non-Linear Latent Tracking for Robust Depth Estimation in Unconstrained Long VideosDVD: Deterministic Video Depth Estimation with Generative Priors🔴Hard