Cross-Domain Zero-Shot Adaptation for Niche Visual Modalities
Adapt vision models to niche domains by pre-training on unlabeled specialized datasets and generating domain-specific pseudo-labels via verifier loops.
Source paper
Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, Christoph Feichtenhofer
Generated by AI. The paper’s authors did not write these.
Adapt vision models to niche domains by pre-training on unlabeled specialized datasets and generating domain-specific pseudo-labels via verifier loops.
Integrate instruction-tuned large language models directly into segmentation architectures to natively process complex linguistic reasoning queries.
Speed up multi-object video tracking in crowded scenes by combining single-pass shared attention with confidence-based masklet pruning.
Extend binary presence tokens into multi-task density heads to improve object counting accuracy from pooled encoder features.
SAM 3 introduces Promptable Concept Segmentation (PCS), a unified task for finding and segmenting all instances of a concept defined by text or image prompts in images and videos. The authors leverage a massive, AI-verified dataset and a decoupled architecture to achieve state-of-the-art results across various open-vocabulary benchmarks.
The model architecture consists of a DETR-based detector and a memory-aware video tracker sharing a common Hiera vision backbone, utilizing a novel 'presence token' to separate global recognition from local object queries. Its performance is driven by a scalable data engine that employs Llama 3.2-based AI verifiers to automate mask quality and exhaustivity checks.