Neural Explicit Memory for Consistent Multi-Agent World Simulation
Augment diffusion transformers with an external spatial memory module to persist environment states and prevent hallucination during long-horizon generation.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
As noted in the Conclusion, the model lacks persistent memory. Because the architecture relies on a sliding context window (autoregressive DiT), when players leave each other's field of view or return to a previously visited location that has slid out of context, the model hallucinates a new state rather than recalling the modified environment.
“Second, our model lacks persistent memory: when players leave each other’s field of view, the model loses track of their shared context, and their trajectories begin to diverge.”
Limitation
The specific weakness is the lack of object permanence and state persistence outside the immediate temporal context window, leading to world-state divergence between agents over long horizons.
Proposed Solution
Augment the DiT architecture with an external, read-write memory module (e.g., a spatial voxel map or object-centric slot attention memory) that persists independent of the context window. When the model generates a frame, it should query this memory for the relevant spatial coordinates to retrieve the previous state of that location, ensuring consistency even after long absences.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
This is critical for long-horizon planning and reasoning. Without persistence, the world model cannot serve as a reliable environment for training agents to perform complex, spatially distributed tasks.
How to Get Started
How to test it: Hypothesis: External spatial slot memory preserves agent object permanence over long absences. Baseline: Standard Solaris model. Metric: Memory VLM judge accuracy. Stop condition: If external memory causes a drop in visual generation FID exceeding 15% without improving long-horizon memory consistency.
- Design an external read-write spatial voxel map or slot-attention memory structure alongside the transformer architecture.
- Implement read and write mechanisms that map generated frames to specific coordinate slots in the persistent memory.
- Integrate memory querying steps into the diffusion transformer inference pipeline for out-of-context locations.
- Benchmark spatial consistency and long-horizon environment memory in multi-agent simulation environments.
Code linked to this paper on Hugging Face: github.com/solaris-wm/solaris
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The paper WorldMem: Long-term Consistent World Simulation with Memory explicitly explores long-term consistent world simulation using memory modules. Therefore, this specific idea has already been investigated in the literature.
- WorldMem: Long-term Consistent World Simulation with Memory · 2025 · 205 citations
We searched OpenAlex for “Augment diffusion transformers with an external spatial memory module to persist environment states and prevent hallucination during long-horizon generation.” and “Neural Explicit Memory for Consistent Multi-Agent World Simulation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Solaris: Building a Multiplayer Video World Model in MinecraftGeorgy Savva, Oscar Michel, Daohan Lu +6 more · 2026
The paper introduces Solaris, a multiplayer video world model for Minecraft capable of simulating consistent multi-view observations for two agents. To achieve this, the authors built a scalable data generation engine (SolarisEngine), collected a large synthetic dataset, and trained a modified Diffusion Transformer (DiT) using a novel "Checkpointed Self Forcing" technique to enable memory-efficient long-horizon autoregressive training.
Related Ideas
- Recurrent Visual Memory for Infinite-Context TrajectoriesMobile-Agent-v3: Foundamental Agents for GUI Automationđź”´Hard
- Compressive Memory for Agentic ContextsGLM-5: from Vibe Coding to Agentic Engineeringđźź Medium
- Hierarchical Sparse Memory for Ultra-Long Horizon ConsistencyWorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingđźź Medium
- Multi-Agent World Modeling: Extending Camera-Conditioned DiTs to Dynamic EntitiesWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representationâ›”Very hard
- Graph-Augmented Episodic Memory for Constraint-Aware Agentic PlanningEnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settingsđź”´Hard