Skip to content

Source paper

OmniGAIA: Towards Native Omni-Modal AI Agents

Xiaoxi Li, Wenxiang Jiao, Jiarui Jin and 8 more

Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Shijian Wang, Guanting Dong, Jiajie Jin, Hao Wang, Yinuo Wang, Ji-Rong Wen, Yuan Lu, Zhicheng Dou

arXiv 2602.22897 · 202646 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces OmniGAIA, a benchmark designed to evaluate native omni-modal AI agents on tasks requiring multi-hop reasoning and tool use across video, audio, and image modalities. Additionally, it proposes OmniAtlas, a foundation agent trained via a novel pipeline involving hindsight-guided tree exploration, trajectory-level supervised fine-tuning with masked supervision, and OmniDPO for fine-grained error correction.

How it works (AI summary)

The authors construct an 'Omni-modal Event Graph' by extracting fine-grained signals from raw media using foundation models, which is then expanded with external tools and fuzzified to generate complex QA pairs. The OmniAtlas agent is trained using a 'Tool-Integrated Reasoning' paradigm that allows it to autonomously interleave internal reasoning with external tool calls (including active perception tools like `read_video`), optimized via Direct Preference Optimization (DPO) on synthesized trajectories.