Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Retrieval-Augmented Policy Context for Scalability

Integrate dense retrieval before the router to dynamically inject only relevant policies into the prompt, removing hard limits on total policy count.

Time

1-2 weeks

Compute level

Single GPU or less

Resources

1x NVIDIA A100 GPU for embedding generation and router inference with a vector database (e.g., FAISS or Chroma).

Prerequisites

transformer architectures, retrieval systems (RAG)

Scope

modify existing code

The Gap

Section 4.2 describes the input prompt x as containing 'the set of all possible route policies C'. This places a hard limit on the number of routing policies (scalable only up to the context window size) and increases latency linearly as the number of policies grows.

Limitation

The current architecture requires injecting every possible policy description into the prompt for every inference pass. This prevents the system from handling large-scale enterprise environments with hundreds or thousands of specific routing policies.

Proposed Solution

Integrate a dense retrieval step (RAG) before the router. Use a lightweight embedding model to retrieve the top-k most relevant policies from a vector store based on the user query, and inject only those k policies into the Arch-Router prompt. This decouples the total number of policies from the inference cost.

Why This Matters

This enables the router to scale to thousands of user-defined policies without hitting context window limits or degrading inference speed, essential for large-scale SaaS platforms.

How to Get Started

How to test it: We hypothesize that a dense retrieval step will maintain routing accuracy while supporting thousands of policies without exceeding context limits. We will compare against full-prompt inclusion using top-1 routing accuracy and prompt token count, stopping if accuracy drops by more than 3% at 1,000 policies.

  1. Set up a vector database using FAISS to index the full set of available routing policies.
  2. Select a lightweight embedding model to encode incoming user queries for similarity matching.
  3. Modify the router inference pipeline to retrieve the top-k most relevant policies and format them into the prompt.
  4. Benchmark inference latency and routing accuracy as the total number of system policies scales significantly.

Code linked to this paper on Hugging Face: github.com/katanemo/archgw

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

The provided literature consists of broad surveys on Retrieval-Augmented Generation (RAG) and federated RAG systems, such as 'Graph Retrieval-Augmented Generation: A Survey' and 'Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG'. None of these listed results cover using dense retrieval to filter and inject specific policies into an architecture router prompt to decouple policy count from inference cost.

We searched OpenAlex for “Integrate dense retrieval before the router to dynamically inject only relevant policies into the prompt, removing hard limits on total policy count.” and “Retrieval-Augmented Policy Context for Scalability” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

Arch-Router: Aligning LLM Routing with Human Preferences

Co Tran, Salman Paracha, Adil Hafeez +1 more · 2025

This paper introduces Arch-Router, a compact 1.5B parameter model designed to route LLM queries based on user-defined natural language policies rather than opaque performance benchmarks. By decoupling intent classification from model assignment and utilizing a synthetic data pipeline with Domain-Action taxonomies, the authors achieve state-of-the-art routing accuracy and reduced latency compared to proprietary models like GPT-4o.

Related Ideas

Is this a good idea?

Share onXLinkedIn