Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

Entity-Disentangled Memory via Segmentation

Integrate entity segmentation models into the memory update pipeline to store cropped character latents separately from background noise.

Time

a couple of months

Compute level

Single GPU or less

Resources

Standard GPU, public dataset (segmentation models)

Prerequisites

computer vision, segmentation, adapter training

Scope

new model architecture

The Gap

The ablation study (Section 4.4) demonstrates that 'naive' full-frame selection includes irrelevant background noise. While StoryMem uses aesthetic filtering, the memory is still a holistic frame representation, mixing characters and backgrounds into a single latent block.

Limitation

The 'bag of keyframes' approach causes entanglement: changing the background or lighting in a new shot is difficult because the memory strongly biases the generation toward the exact pixel distribution of previous frames, including the old background.

Proposed Solution

Integrate an entity-segmentation module (e.g., SAM or Grounding DINO) into the memory update step. Instead of storing full frames, detect and crop distinct entities (characters, key objects) into separate memory slots. During generation, inject these entity-specific latents via a decoupled cross-attention adapter (similar to IP-Adapter), allowing the model to place consistent characters into entirely new backgrounds.

Why This Matters

This allows for true 'character consistency' independent of the scene, enabling narratives where characters travel between vastly different environments without background leakage.

How to Get Started

  1. Integrate an open-source segmentation model like Grounding DINO or SAM into the preprocessing pipeline.
  2. Modify the memory bank to store object-specific latent slots instead of full-frame representations.
  3. Implement a decoupled cross-attention adapter to inject entity latents during generation.
  4. Evaluate background blending and character consistency on new composition benchmarks.

Code linked to this paper on Hugging Face: github.com/Kevin-thu/StoryMem

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation directly targets entity-centric memory and consistency for video generation. However, it leaves open the specific integration of segmentation modules like SAM to crop distinct entities into decoupled cross-attention adapters.

We searched OpenAlex for “Integrate entity segmentation models into the memory update pipeline to store cropped character latents separately from background noise.” and “Entity-Disentangled Memory via Segmentation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

StoryMem: Multi-shot Long Video Storytelling with Memory

Kaiwen Zhang, Liming Jiang, Angtian Wang +6 more · 2025

The paper introduces StoryMem, a framework that enables pre-trained single-shot video diffusion models to generate coherent, multi-shot long videos. It employs a Memory-to-Video (M2V) mechanism that stores and retrieves visual keyframes from previous shots to maintain narrative consistency across minute-long sequences.

Related Ideas

Is this a good idea?

Share onXLinkedIn