Skip to content

Source paper

DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation

Xu Guo, Fulong Ye, Qichao Sun and 7 more

Xu Guo, Fulong Ye, Qichao Sun, Liyang Chen, Bingchuan Li, Pengze Zhang, Jiawei Liu, Songtao Zhao, Qian He, Xiangwang Hou

arXiv 2602.12160 · 202633 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper presents DreamID-Omni, a unified generative framework capable of handling three human-centric tasks—reference-based audio-video generation (R2AV), video editing (RV2AV), and audio-driven animation (RA2V)—within a single model. By treating these tasks as variations of conditional generation, the authors achieve state-of-the-art performance in multi-speaker scenarios while ensuring consistent identity and voice timbre control.

How it works (AI summary)

The model is built on a Symmetric Conditional Diffusion Transformer (DiT) featuring dual streams for audio and video that interact via bidirectional cross-attention. It employs a 'Syn-RoPE' mechanism to bind identities to specific temporal segments at the signal level, and 'Structured Captions' with anchor tokens to disentangle semantic attributes, all trained via a Multi-Task Progressive Training curriculum.