Entity-Disentangled Memory via Segmentation
Integrate entity segmentation models into the memory update pipeline to store cropped character latents separately from background noise.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The ablation study (Section 4.4) demonstrates that 'naive' full-frame selection includes irrelevant background noise. While StoryMem uses aesthetic filtering, the memory is still a holistic frame representation, mixing characters and backgrounds into a single latent block.
Limitation
The 'bag of keyframes' approach causes entanglement: changing the background or lighting in a new shot is difficult because the memory strongly biases the generation toward the exact pixel distribution of previous frames, including the old background.
Proposed Solution
Integrate an entity-segmentation module (e.g., SAM or Grounding DINO) into the memory update step. Instead of storing full frames, detect and crop distinct entities (characters, key objects) into separate memory slots. During generation, inject these entity-specific latents via a decoupled cross-attention adapter (similar to IP-Adapter), allowing the model to place consistent characters into entirely new backgrounds.
Why This Matters
This allows for true 'character consistency' independent of the scene, enabling narratives where characters travel between vastly different environments without background leakage.
How to Get Started
- Integrate an open-source segmentation model like Grounding DINO or SAM into the preprocessing pipeline.
- Modify the memory bank to store object-specific latent slots instead of full-frame representations.
- Implement a decoupled cross-attention adapter to inject entity latents during generation.
- Evaluate background blending and character consistency on new composition benchmarks.
Code linked to this paper on Hugging Face: github.com/Kevin-thu/StoryMem
Suggested by AI. Adapt it to your setup.
Has This Been Done?
EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation directly targets entity-centric memory and consistency for video generation. However, it leaves open the specific integration of segmentation modules like SAM to crop distinct entities into decoupled cross-attention adapters.
- EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation · 2026 · 5 citations
We searched OpenAlex for “Integrate entity segmentation models into the memory update pipeline to store cropped character latents separately from background noise.” and “Entity-Disentangled Memory via Segmentation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
StoryMem: Multi-shot Long Video Storytelling with MemoryKaiwen Zhang, Liming Jiang, Angtian Wang +6 more · 2025
The paper introduces StoryMem, a framework that enables pre-trained single-shot video diffusion models to generate coherent, multi-shot long videos. It employs a Memory-to-Video (M2V) mechanism that stores and retrieves visual keyframes from previous shots to maintain narrative consistency across minute-long sequences.
Related Ideas
- Persistent Identity Memory ModuleStable Video Infinity: Infinite-Length Video Generation with Error Recycling🟡Fairly easy
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Automated Ablation Study for Component IsolationMIRIX-2.0: Multi-Agent Memory System🟡Fairly easy
- Adaptive Attention-Based Memory CompressionHelios: Real Real-Time Long Video Generation Model🟠Medium
- Omnimodal Conflict Resolution in Memory EvolutionMemory in the Age of AI Agents🔴Hard