Retrieval-Augmented Decoding for Low-Parameter Hallucination Reduction
Integrate a lightweight token-level retrieval-augmented generation module into low-parameter vision models to reduce hallucinations of rare entities without full retraining.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The core recognition module relies on ERNIE-4.5-0.3B (Section 2.1). While efficient, 0.3B parameter models suffer significantly higher hallucination rates compared to 7B+ models when dealing with ambiguous visual inputs or rare entities, as they lack extensive parametric world knowledge.
Limitation
The 0.3B backbone is prone to 'hallucinations' when recognizing obscure named entities or messy handwriting, as it cannot rely on internal knowledge to auto-correct ambiguous OCR results.
Proposed Solution
Integrate a lightweight, token-level Retrieval-Augmented Generation (RAG) module during the inference phase. Instead of relying solely on the 0.3B model's weights, implementing a k-Nearest Neighbors (kNN) look-up against a domain-specific dictionary or corpus during decoding could ground the generation of rare entities without retraining the VLM.
Why This Matters
This allows the ultra-compact model to punch above its weight class in terms of accuracy on technical or domain-specific documents (medical, legal) without increasing model size.
How to Get Started
- Build a domain-specific text corpus and index it using a fast k-Nearest Neighbor search library like FAISS.
- Modify the decoding loop of a small Vision-Language Model to query the external index at each token generation step.
- Tune the interpolation weight between the model's vocabulary logits and the retrieved token probabilities.
- Evaluate hallucination rates on rare entities using a specialized document understanding benchmark.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Several listed works discuss retrieval-augmented generation and hallucination reduction, such as 'Reducing hallucination in structured outputs via Retrieval-Augmented Generation' and 'Scalable Token-Level Hallucination Detection in Large Language Models'. However, none of these specific references evaluate the exact combination of a token-level kNN retrieval module during decoding specifically tailored for low-parameter models as proposed.
- Scalable Token-Level Hallucination Detection in Large Language Models · 2026 · 1 citations
- Reducing hallucination in structured outputs via Retrieval-Augmented Generation · 2024 · 315 citations
- Retrieval Augmentation Reduces Hallucination in Conversation · 2021 · 1229 citations
We searched OpenAlex for “Integrate a lightweight token-level retrieval-augmented generation module into low-parameter vision models to reduce hallucinations of rare entities without full retraining.” and “Retrieval-Augmented Decoding for Low-Parameter Hallucination Reduction” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language ModelCheng Cui, Ting Sun, Suyin Liang +15 more · 2025
The paper presents PaddleOCR-VL, a resource-efficient document parsing system built around a 0.9B parameter Vision-Language Model (VLM). By decoupling layout analysis (using PP-DocLayoutV2) from element recognition (using the VLM with a NaViT-style encoder and ERNIE-4.5-0.3B backbone), the authors achieve SOTA performance on multilingual document benchmarks while maintaining low latency suitable for deployment.
Related Ideas
- Differentiable Relaxation of Written Associative Memories for Few-Shot Fine-TuningWeight-Space Programming Doesn't Need Transformers🟡Fairly easy
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard
- Contrastive Modality-Aware Re-ranking to Mitigate Text BiasRAG-Anything: All-in-One RAG Framework🟡Fairly easy
- Adaptive Token-Level Gating for Heterogeneous Modality AlignmentInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🟡Fairly easy
- Curriculum Bit-Depth Training for Lightweight ModelsBitDance: Scaling Autoregressive Generative Models with Binary Tokens🟡Fairly easy