Skip to content

Source paper

VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos

Xubin Ren, Lingrui Xu, Long Xia and 3 more

Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, Chao Huang

arXiv 2502.01549 · 20252 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper presents VideoRAG, a Retrieval-Augmented Generation framework designed to process and understand extremely long-context videos by combining graph-based textual knowledge grounding with multi-modal context encoding. The system indexes videos by converting visual and audio content into a structured knowledge graph while simultaneously maintaining visual embeddings, enabling Large Language Models to retrieve and synthesize information across multiple lengthy videos for tasks like question answering.

How it works (AI summary)

The framework utilizes a dual-channel indexing system: (1) Graph-based Textual Knowledge Grounding, where videos are segmented and processed by VLMs and ASR to generate text that is structured into a Knowledge Graph using LLMs; and (2) Multi-Modal Context Encoding, which uses ImageBind to create embeddings for video clips. Retrieval is performed via a hybrid approach that combines semantic entity matching within the graph and visual similarity search, followed by an LLM-based filtering and re-captioning stage.