Hard-Negative Mining for Visual Grounding
Train audio-visual grounding models using off-screen sound events as hard negatives to effectively disentangle visual presence from acoustic activity.
Source paper
Bowen Shi, Andros Tjandra, John Hoffman, Helin Wang, Yi-Chiao Wu, Luya Gao, Julius Richter, Matt Le, Apoorv Vyas, Sanyuan Chen, Christoph Feichtenhofer, Piotr Dollár, Wei-Ning Hsu, Ann Lee
Generated by AI. The paper’s authors did not write these.
Train audio-visual grounding models using off-screen sound events as hard negatives to effectively disentangle visual presence from acoustic activity.
Distill a multi-step diffusion audio model into a 1-step or 2-step student network using consistency distillation to enable real-time inference.
Replace windowed multi-diffusion with Ring Attention to process long audio sequences natively without quadratic memory scaling.
This paper presents SAM Audio, a foundation model for universal audio source separation that unifies text, visual, and temporal span prompting within a single framework. By leveraging large-scale training on speech, music, and general sounds, the model achieves state-of-the-art performance across diverse benchmarks, supported by a new evaluation suite and a reference-free judge model.
The model employs a Diffusion Transformer (DiT) architecture trained with Flow Matching, operating on compressed latent representations generated by a DAC-VAE audio codec, and conditions on multimodal prompts (T5 text embeddings, visual features, and temporal tokens).