Contrastive Modality-Aware Re-ranking to Mitigate Text Bias
Train a contrastive cross-encoder to re-rank retrieved text and image candidates based on modal necessity and resolve conflicts caused by text-centric retrieval bias.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In Appendix A.5 (Case 1), the authors identify a 'Text-Centric Retrieval Bias,' where the system retrieves noisy textual evidence instead of the correct visual answer (e.g., a specific flow in a diagram). The current 'modality-aware query encoding' (Section 2.3) relies on lexical cues and query analysis which fails when text and visual signals conflict or when the query implies visual intent without explicit keywords.
Limitation
The retrieval mechanism systematically prioritizes textual passages over visual nodes when they share keywords, leading to failures in answering questions that strictly require reading a chart or diagram (e.g., Figure 11 failure case).
Proposed Solution
Implement a Contrastive Modality-Alignment Re-ranker. Train a lightweight cross-encoder that takes the query and a candidate (text or image) and scores them not just on semantic similarity, but on 'modal necessity.' Specifically, introduce a 'Visual-Text Disagreement' module: if retrieved text and image descriptions conflict, the system should prioritize the modality predicted to be more authoritative for that specific query type (e.g., 'spatial flow' queries favor image nodes).
Why This Matters
Addressing this bias is crucial for trust in multimodal RAG; currently, the system can be 'actively misled' by text when the ground truth lies in a visual artifact.
How to Get Started
- Curate multimodal evaluation datasets such as DocBench featuring ambiguous query types.
- Implement a cross-encoder architecture that evaluates query-candidate pairs for semantic similarity and modal necessity.
- Build a modality disagreement detection module to prioritize visual or textual evidence based on query intent.
- Benchmark retrieval accuracy against standard lexical and query-analysis baselines.
Code linked to this paper on Hugging Face: github.com/HKUDS/RAG-Anything
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers address contrastive modality-aware re-ranking or visual-text disagreement mitigation for text bias. Works like Learnable Pillar-based Re-ranking for Image-Text Retrieval and Region-R1 focus on other re-ranking architectures, while Modality-Aware Bias Mitigation addresses person re-identification. Thus, the specific proposal remains unexplored among these results.
We searched OpenAlex for “Train a contrastive cross-encoder to re-rank retrieved text and image candidates based on modal necessity and resolve conflicts caused by text-centric retrieval bias.” and “Contrastive Modality-Aware Re-ranking to Mitigate Text Bias” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
RAG-Anything: All-in-One RAG FrameworkZirui Guo, Xubin Ren, Lingrui Xu +2 more · 2025
This paper introduces RAG-Anything, a multimodal Retrieval-Augmented Generation framework designed to treat non-textual elements (images, tables, equations) as first-class knowledge entities alongside text. It overcomes the limitations of text-centric RAG systems by constructing a unified dual-graph (cross-modal and text-based) that preserves structural relationships and visual semantics, enabling precise reasoning over heterogeneous documents.
Related Ideas
- Text-Guided Multimodal Memory RetrievalStoryMem: Multi-shot Long Video Storytelling with Memory🟠Medium
- Native Multimodal Knowledge Graph IntegrationVideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos🟠Medium
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard
- Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR RecognitionV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟡Fairly easy
- Adaptive Token-Level Gating for Heterogeneous Modality AlignmentInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🟡Fairly easy