Text-Guided Multimodal Memory Retrieval
Add a cross-attention layer to the memory bank that queries text embeddings to prevent identity swaps in multi-character video generation.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Appendix C highlights that the current memory update function is purely visual and discards textual information from previous prompts. Consequently, in multi-character scenarios (e.g., Fig. 7), the model fails to map specific prompt entities (e.g., 'Sheldon') to the correct visual memory, leading to identity swaps.
Limitation
The retrieval mechanism is ambiguous because it relies solely on visual similarity, lacking the semantic labels necessary to distinguish between different characters or objects stored in the memory bank.
Proposed Solution
Implement a multimodal memory bank that stores pairs of (visual_latent, text_embedding) for each keyframe. Modify the attention mechanism in the DiT to include a cross-attention layer where the current shot's text prompt queries the stored text embeddings. Use the resulting attention weights to selectively modulate the injection of visual memory latents, ensuring the model attends to the specific character or object requested by the text.
Why This Matters
This would solve the critical failure mode of identity confusion in complex, multi-subject narratives, significantly improving the model's utility for detailed storytelling.
How to Get Started
- Modify the memory bank data structure to store text-visual latent pairs for keyframes.
- Implement a text-to-visual cross-attention module inside the diffusion transformer blocks.
- Fine-tune the modified model using a multi-character video dataset.
- Evaluate identity consistency metrics against baseline models without textual memory querying.
Code linked to this paper on Hugging Face: github.com/Kevin-thu/StoryMem
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed results implement a multimodal memory bank with text-queried cross-attention for DiT-based video generation. While papers like 'Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding' and 'Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence' explore retrieval or memory for understanding tasks, they do not address modulating visual latent injection via text queries in diffusion transformers.
- Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence · 2026 · 0 citations
- Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding · 2025 · 46 citations
We searched OpenAlex and arXiv for “Add a cross-attention layer to the memory bank that queries text embeddings to prevent identity swaps in multi-character video generation.” and “Text-Guided Multimodal Memory Retrieval” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
StoryMem: Multi-shot Long Video Storytelling with MemoryKaiwen Zhang, Liming Jiang, Angtian Wang +6 more · 2025
The paper introduces StoryMem, a framework that enables pre-trained single-shot video diffusion models to generate coherent, multi-shot long videos. It employs a Memory-to-Video (M2V) mechanism that stores and retrieves visual keyframes from previous shots to maintain narrative consistency across minute-long sequences.
Related Ideas
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard
- Persistent Identity Memory ModuleStable Video Infinity: Infinite-Length Video Generation with Error Recycling🟡Fairly easy
- Contrastive Modality-Aware Re-ranking to Mitigate Text BiasRAG-Anything: All-in-One RAG Framework🟡Fairly easy
- Spatially-Guided Attention Masks for Multi-Speaker DisentanglementLTX-2: Efficient Joint Audio-Visual Foundation Model🔴Hard
- Omnimodal Conflict Resolution in Memory EvolutionMemory in the Age of AI Agents🔴Hard