Spatially-Guided Attention Masks for Multi-Speaker Disentanglement
A method that applies spatial cross-attention masks derived from face tracking to bind distinct audio streams to specific speakers in multi-speaker video generation.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
As noted in Section 7, the model struggles in multi-speaker scenarios, often assigning spoken content to the wrong character. The current cross-attention mechanism aligns global audio features with global video features, lacking the spatial granularity to bind specific audio waveforms (voice A) to specific visual regions (face A) explicitly.
Limitation
Confusion in assigning specific dialogue lines to the correct speaker in scenes with multiple characters.
Proposed Solution
Introduce spatial cross-attention masking. During training/inference, use an object detector or face tracker to generate spatial masks for each character. Modify the text conditioning to associate specific dialogue segments with these spatial masks, forcing the audio stream to attend specifically to the visual tokens within the active speaker's bounding box during the cross-modal attention layers.
Why This Matters
Solving the 'cocktail party problem' in generation is essential for creating realistic dialogue scenes, which are fundamental to cinema and complex content creation.
How to Get Started
- Integrate an off-the-shelf face tracker or object detector, such as YOLOv8 or RetinaFace, into the video-audio multi-speaker data pipeline.
- Construct spatial cross-attention masks corresponding to each character's bounding box across video frames.
- Modify the cross-modal attention layers of the baseline model to restrict audio-text feature conditioning to the active speaker's spatial mask region.
- Evaluate speaker disentanglement and lip-sync accuracy on multi-speaker audiovisual datasets like LRS3 or ActivityNet.
Code linked to this paper on Hugging Face: github.com/Lightricks/LTX-2
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as AnyTalker and CineDance discuss multi-person talking video generation and narrative control, while Visually-Guided Audio Spatialization addresses spatial conditioning. They touch upon multi-speaker interactions and spatial awareness, but do not specifically employ spatial cross-attention masking using object bounding boxes to link dialogue to speaker visual tokens.
- CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generation · 2026 · 8 citations
- AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement · 2025 · 22 citations
- Visually-Guided Audio Spatialization in Video with Geometry-Aware Multi-task Learning · 2023 · 50 citations
We searched OpenAlex for “A method that applies spatial cross-attention masks derived from face tracking to bind distinct audio streams to specific speakers in multi-speaker video generation.” and “Spatially-Guided Attention Masks for Multi-Speaker Disentanglement” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
LTX-2: Efficient Joint Audio-Visual Foundation ModelYoav HaCohen, Benny Brazowski, Nisan Chiprut +26 more · 2026
This paper presents LTX-2, an efficient, open-source multimodal foundation model capable of jointly generating synchronized video and audio from text prompts. By utilizing an asymmetric dual-stream transformer architecture with bidirectional cross-attention, the model achieves state-of-the-art audiovisual alignment and inference speed, overcoming the latency and coherence issues found in sequential (video-then-audio) generation pipelines.
Related Ideas
- Dynamic Attention-Based Identity RoutingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟡Fairly easy
- Latent Visual-Audio Contrastive BindingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟠Medium
- Interaction-Aware Multi-Subject ModelingOmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions🟠Medium
- Hierarchical Disentanglement for Controllable TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🟠Medium
- Text-Guided Multimodal Memory RetrievalStoryMem: Multi-shot Long Video Storytelling with Memory🟠Medium