Latent Space Disentanglement for Robust Global Style Transfer in Real-World Grounded Video Generation
A method that combines Low-Rank Adaptation with cross-attention control to explicitly disentangle spatial geometry conditioning from global style conditioning in real-world world models.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
SWM fine-tunes a base pretrained world model on real-world street-view images, which predominantly feature standard daytime lighting and weather conditions.
“However, prompts involving global style changes such as day-to-night transitions or weather variations tend to be less faithfully followed after fine-tuning on street-view data.”
Limitation
The model exhibits catastrophic forgetting of the base model's ability to handle global style changes (e.g., day-to-night transitions, severe weather conditions). To compensate, the authors currently have to artificially augment the training data using a computationally expensive external video stylization pipeline.
Proposed Solution
Implement a Low-Rank Adaptation (LoRA) module combined with a cross-attention control mechanism that explicitly disentangles spatial geometry conditioning (from street-view images) from global style conditioning (from text prompts). Train the base spatial layout weights on the street-view data while strictly regularizing the style-mapping layers to preserve the pre-trained model's stylistic and atmospheric priors.
Why This Matters
This removes the reliance on synthetic, offline video stylization pipelines during training, allowing the model to generate highly diverse, zero-shot environmental variations seamlessly, which is critical for edge-case simulation.
How to Get Started
How to test it: Hypothesis: LoRA with regularized cross-attention disentangling spatial layout from global style preserves the pretrained model's atmospheric priors without needing the 10K external stylized video augmentations. Compare against SWM (TF) trained with stylized video augmentation using text-style alignment (CLIP/VLM/human preference) on global style prompts and FID, FVD, mPSNR, mLPIPS on Busan-City-Bench and Ann-Arbor-City-Bench; stop if style adherence underperforms the stylized-augmentation baseline or mPSNR degrades by >0.5 dB.
- Set up a pretrained world model codebase capable of conditioning on real-world street-view image inputs.
- Implement a Low-Rank Adaptation (LoRA) module coupled with cross-attention control to separate layout and style pathways.
- Train spatial layout weights on real-world driving datasets while applying regularization constraints to style-mapping layers.
- Evaluate generated video samples across diverse text prompts to ensure stylistic priors are preserved without degrading geometric accuracy.
Code linked to this paper on Hugging Face: github.com/naver-ai/seoul-world-model
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as "Attention Calibration for Disentangled Text-to-Image Personalization" and "Disco: Disentangled Control for Realistic Human Dance Generation" discuss disentangled control and latent space adjustments, but none explicitly detail the combination of LoRA and cross-attention control to isolate spatial geometry conditioning from global style conditioning for grounded video generation.
- Disco: Disentangled Control for Realistic Human Dance Generation · 2024 · 267 citations
- Attention Calibration for Disentangled Text-to-Image Personalization · 2024 · 56 citations
We searched OpenAlex for “A method that combines Low-Rank Adaptation with cross-attention control to explicitly disentangle spatial geometry conditioning from global style conditioning in real-world world models.” and “Latent Space Disentanglement for Robust Global Style Transfer in Real-World Grounded Video Generation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Grounding World Simulation Models in a Real-World MetropolisJunyoung Seo, Hyunwook Choi, Minkyung Kwon +10 more · 2026
The paper introduces the Seoul World Model (SWM), a city-scale video generative world model that anchors simulated environments to the actual physical layout of a real metropolis. The main contribution is a retrieval-augmented generation framework that leverages cross-temporal pairing, synthetic urban data, and a Virtual Lookahead Sink to synthesize spatially faithful, temporally consistent, and long-horizon videos.
Related Ideas
- Style-Invariant Error RecyclingStable Video Infinity: Infinite-Length Video Generation with Error Recycling🟡Fairly easy
- End-to-End Layered Generation to Replace Heuristic CompositingDreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformer🔴Hard
- Natural Predictive Pre-training on Web-Scale VideoIn Pursuit of Pixel Supervision for Visual Pre-training⛔Very hard
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation⛔Very hard
- Scaling Unified Multimodal Models via Large Pre-trained Vision-Language InitializationsCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation🔴Hard