Hard-Negative Mining for Visual Grounding
Train audio-visual grounding models using off-screen sound events as hard negatives to effectively disentangle visual presence from acoustic activity.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Visual-prompted separation significantly underperforms text prompting (Section 7.2) and struggles with off-screen sounds where the visual mask exists but the sound does not, or vice versa (Conclusion).
Limitation
The current training strategy does not explicitly penalize the model for associating visual cues with off-screen sounds, leading to lower performance (2.61 OVR vs 3.50 OVR for text) and reliance on noisy audio-visual correspondence.
Proposed Solution
Implement a contrastive learning curriculum that specifically mines 'off-screen' sound events as hard negatives. This involves augmenting the training pipeline to include video frames where the object is visible but silent, forcing the model to disentangle visual presence from acoustic activity.
Why This Matters
Visual prompting is currently the weakest modality; fixing this would enable robust applications in film editing and AR glasses where text prompts are impractical.
How to Get Started
- Construct a balanced contrastive dataset containing video clips where sound sources are explicitly out of frame.
- Modify the audio-visual alignment loss function to incorporate hard-negative penalty terms for silent visible objects.
- Finetune a baseline audio-visual grounding model using public repositories on datasets like VGGSound.
- Evaluate localization performance on challenging scenarios involving ambiguous off-screen audio sources.
Code linked to this paper on Hugging Face: github.com/facebookresearch/sam-audio
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers address hard-negative mining using off-screen sound events for visual grounding or contrastive learning curriculums for sound source localization. While papers like Hearing and Seeing Through CLIP discuss self-supervised sound source localization broadly, they do not cover this specific training augmentation strategy.
- Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization · 2026 · 4 citations
We searched OpenAlex for “Train audio-visual grounding models using off-screen sound events as hard negatives to effectively disentangle visual presence from acoustic activity.” and “Hard-Negative Mining for Visual Grounding” and an AI compared the idea with the top 5 results. A small search can miss relevant work.
Source Paper
SAM Audio: Segment Anything in AudioBowen Shi, Andros Tjandra, John Hoffman +11 more · 2025
This paper presents SAM Audio, a foundation model for universal audio source separation that unifies text, visual, and temporal span prompting within a single framework. By leveraging large-scale training on speech, music, and general sounds, the model achieves state-of-the-art performance across diverse benchmarks, supported by a new evaluation suite and a reference-free judge model.
Related Ideas
- Latent Visual-Audio Contrastive BindingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟠Medium
- Spatially-Guided Attention Masks for Multi-Speaker DisentanglementLTX-2: Efficient Joint Audio-Visual Foundation Model🔴Hard
- Internal Mask Generation for Precise LocalizationUniVideo: Unified Understanding, Generation, and Editing for Videos🟡Fairly easy
- Multimodal Skill Encoding for Visual EnvironmentsSkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning🔴Hard
- Token-Based Paralinguistic InjectionIndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System🟡Fairly easy