Scalable Multi-View World Modeling via Sparse Interaction Attention
Replace global player attention with a sparse or graph-based mechanism to scale multi-agent world simulations to ten or more players efficiently.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The current architecture uses a 'Shared Self-Attention' block (Fig. 5) that attends across all tokens of all players. While the paper restricts P=2 (Section 4.1), the computational cost of full self-attention scales quadratically with the number of players (O(P2)), making it computationally prohibitive to simulate 5+ or 10+ agents.
“We introduce Solaris, a multiplayer video world model that simulates consistent multi-view observations. To enable this, we develop a multiplayer data system designed for robust, continuous, and automated data collection on video games such as Minecraft.”
Limitation
The specific weakness is the quadratic computational complexity of the shared attention mechanism, which limits the model's scalability to a small number of concurrent agents.
Proposed Solution
Replace the global shared attention with a sparse or graph-based attention mechanism where agents only attend to other agents within a specific proximity or visual line-of-sight. Alternatively, implement a hierarchical attention scheme where a global low-resolution state enforces consensus while local high-resolution models handle individual views.
Why This Matters
Real-world environments and complex games involve many agents. Scaling the world model to N > 2 is essential for simulating realistic social dynamics and crowd behaviors.
How to Get Started
How to test it: Hypothesis: Sparse attention reduces scaling complexity while maintaining multi-view consistency. Baseline: Full multiplayer self-attention DiT. Metric: VLM consistency score and throughput (frames per second). Stop condition: If sparse attention degrades consistency accuracy by more than 5% compared to full attention.
- Audit existing shared self-attention layers in the baseline world model codebase to profile memory bottlenecks at scale.
- Implement a distance-based or line-of-sight adjacency matrix to restrict attention weights between interacting agents.
- Benchmark the modified sparse attention architecture against the full self-attention baseline using synthetic multi-agent trajectories.
- Evaluate simulation fidelity and computational savings as the number of active players scales from 2 to 10.
Code linked to this paper on Hugging Face: github.com/solaris-wm/solaris
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers combine sparse or graph-based attention with multi-view world modeling as proposed here. The search results discuss general multi-agent reinforcement learning or sparse attention for tabular and CT data, leaving this specific architectural proposal open.
We searched OpenAlex for “Replace global player attention with a sparse or graph-based mechanism to scale multi-agent world simulations to ten or more players efficiently.” and “Scalable Multi-View World Modeling via Sparse Interaction Attention” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Solaris: Building a Multiplayer Video World Model in MinecraftGeorgy Savva, Oscar Michel, Daohan Lu +6 more · 2026
The paper introduces Solaris, a multiplayer video world model for Minecraft capable of simulating consistent multi-view observations for two agents. To achieve this, the authors built a scalable data generation engine (SolarisEngine), collected a large synthetic dataset, and trained a modified Diffusion Transformer (DiT) using a novel "Checkpointed Self Forcing" technique to enable memory-efficient long-horizon autoregressive training.
Related Ideas
- Multi-Entity Action-State Modeling for Interactive EnvironmentsWorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling🔴Hard
- Multi-Agent World Modeling: Extending Camera-Conditioned DiTs to Dynamic EntitiesWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation⛔Very hard
- Complex Social Interaction BenchmarkingVery Large-Scale Multi-Agent Simulation in AgentScope🟠Medium
- Latency-Aware Scalable Video Tracking for Crowded EnvironmentsSAM 3: Segment Anything with Concepts🟠Medium
- Interaction-Aware Multi-Subject ModelingOmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions🟠Medium