Dynamic Attention-Based Identity Routing
Deploy dynamic cross-identity attention masking to remove fixed positional embedding margins and support arbitrary speakers and sequence lengths.
Source paper
Xu Guo, Fulong Ye, Qichao Sun, Liyang Chen, Bingchuan Li, Pengze Zhang, Jiawei Liu, Songtao Zhao, Qian He, Xiangwang Hou
Generated by AI. The paper’s authors did not write these.
Deploy dynamic cross-identity attention masking to remove fixed positional embedding margins and support arbitrary speakers and sequence lengths.
Incorporate a contrastive attention loss to bind audio generation streams directly to correct visual identities without relying on text prompt anchor tokens.
Apply modality-specific rectified flow matching with independent velocity fields to optimize joint audio-video generation paths simultaneously.
This paper presents DreamID-Omni, a unified generative framework capable of handling three human-centric tasks—reference-based audio-video generation (R2AV), video editing (RV2AV), and audio-driven animation (RA2V)—within a single model. By treating these tasks as variations of conditional generation, the authors achieve state-of-the-art performance in multi-speaker scenarios while ensuring consistent identity and voice timbre control.
The model is built on a Symmetric Conditional Diffusion Transformer (DiT) featuring dual streams for audio and video that interact via bidirectional cross-attention. It employs a 'Syn-RoPE' mechanism to bind identities to specific temporal segments at the signal level, and 'Structured Captions' with anchor tokens to disentangle semantic attributes, all trained via a Multi-Task Progressive Training curriculum.