Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchNo close match found

Hard-Negative Mining for Visual Grounding

Train audio-visual grounding models using off-screen sound events as hard negatives to effectively disentangle visual presence from acoustic activity.

Time

a couple of months

Compute level

Single GPU or less

Resources

Public audio-visual datasets (e.g., VGGSound, AudioSet)

Prerequisites

Computer vision, contrastive learning, multimodal alignment

Scope

modify existing code

The Gap

Visual-prompted separation significantly underperforms text prompting (Section 7.2) and struggles with off-screen sounds where the visual mask exists but the sound does not, or vice versa (Conclusion).

Limitation

The current training strategy does not explicitly penalize the model for associating visual cues with off-screen sounds, leading to lower performance (2.61 OVR vs 3.50 OVR for text) and reliance on noisy audio-visual correspondence.

Proposed Solution

Implement a contrastive learning curriculum that specifically mines 'off-screen' sound events as hard negatives. This involves augmenting the training pipeline to include video frames where the object is visible but silent, forcing the model to disentangle visual presence from acoustic activity.

Why This Matters

Visual prompting is currently the weakest modality; fixing this would enable robust applications in film editing and AR glasses where text prompts are impractical.

How to Get Started

  1. Construct a balanced contrastive dataset containing video clips where sound sources are explicitly out of frame.
  2. Modify the audio-visual alignment loss function to incorporate hard-negative penalty terms for silent visible objects.
  3. Finetune a baseline audio-visual grounding model using public repositories on datasets like VGGSound.
  4. Evaluate localization performance on challenging scenarios involving ambiguous off-screen audio sources.

Code linked to this paper on Hugging Face: github.com/facebookresearch/sam-audio

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 5 results

None of the listed papers address hard-negative mining using off-screen sound events for visual grounding or contrastive learning curriculums for sound source localization. While papers like Hearing and Seeing Through CLIP discuss self-supervised sound source localization broadly, they do not cover this specific training augmentation strategy.

We searched OpenAlex for “Train audio-visual grounding models using off-screen sound events as hard negatives to effectively disentangle visual presence from acoustic activity.” and “Hard-Negative Mining for Visual Grounding” and an AI compared the idea with the top 5 results. A small search can miss relevant work.

Source Paper

SAM Audio: Segment Anything in Audio

Bowen Shi, Andros Tjandra, John Hoffman +11 more · 2025

This paper presents SAM Audio, a foundation model for universal audio source separation that unifies text, visual, and temporal span prompting within a single framework. By leveraging large-scale training on speech, music, and general sounds, the model achieves state-of-the-art performance across diverse benchmarks, supported by a new evaluation suite and a reference-free judge model.

Related Ideas

Is this a good idea?

Share onXLinkedIn