Uncertainty-Aware Active Routing for Ambiguous Policies
Quantify output uncertainty during policy routing to trigger clarification questions when incoming queries match overlapping or ambiguous policies.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 6.3 states that 'routing accuracy is constrained by the precision of the policy set; ambiguous or overlapping policy descriptions directly degrade routing performance.' The current Cross-Entropy training objective forces the model to make a hard decision between overlapping policies (e.g., 'review contract' vs 'legal analysis').
“First, routing accuracy is constrained by the precision of the policy set; ambiguous or overlapping policy descriptions directly degrade routing performance.”
Limitation
The model lacks a mechanism to handle ambiguity. When user-defined policies overlap, the router acts deterministically (likely hallucinating certainty), rather than asking for clarification or merging policies.
Proposed Solution
Implement uncertainty quantification (e.g., measuring entropy of the output logits over route tokens). If the entropy exceeds a threshold, the router should output a special 'AMBIGUOUS' token or generate a clarification question prompting the user to distinguish between the competing policies. This requires fine-tuning on a new dataset slice where the 'correct' label is a clarification request when policies overlap.
Why This Matters
This prevents 'silent failures' where the router confidently sends a query to the wrong expert due to poor user descriptions, significantly improving robustness in real-world deployments.
How to Get Started
How to test it: We hypothesize that measuring logit entropy will reliably detect overlapping policies and prompt for clarification. We will compare against a deterministic routing baseline using classification accuracy and clarification appropriateness, stopping if ambiguity detection precision falls below 85%.
- Extract output logits from a baseline routing model and calculate entropy across candidate policy tokens.
- Synthesize a dataset slice containing intentionally ambiguous queries where the correct label is a clarification request.
- Fine-tune the routing policy to output a special ambiguous token when entropy exceeds a defined threshold.
- Measure the reduction in misclassification errors on overlapping policy benchmarks.
Code linked to this paper on Hugging Face: github.com/katanemo/archgw
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers address uncertainty-aware active routing for ambiguous model policies using clarification tokens and fine-tuning on a specialized dataset slice. While papers like 'Learning When to Collaborate: Selective Multi-Agent Medical Reasoning via Uncertainty-Aware Routing' touch on uncertainty routing, they do not implement the specific ambiguity-handling mechanism proposed here.
- Learning When to Collaborate: Selective Multi-Agent Medical Reasoning via Uncertainty-Aware Routing · 2026 · 0 citations
We searched OpenAlex for “Quantify output uncertainty during policy routing to trigger clarification questions when incoming queries match overlapping or ambiguous policies.” and “Uncertainty-Aware Active Routing for Ambiguous Policies” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Arch-Router: Aligning LLM Routing with Human PreferencesCo Tran, Salman Paracha, Adil Hafeez +1 more · 2025
This paper introduces Arch-Router, a compact 1.5B parameter model designed to route LLM queries based on user-defined natural language policies rather than opaque performance benchmarks. By decoupling intent classification from model assignment and utilizing a synthetic data pipeline with Domain-Action taxonomies, the authors achieve state-of-the-art routing accuracy and reduced latency compared to proprietary models like GPT-4o.
Related Ideas
- Neuro-Symbolic Constraint Enforcement ModuleMobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios🟠Medium
- Domain-Grounded Test-Time Compute Routing for Policy WorkflowsEnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings🟡Fairly easy
- Proactive User-in-the-Loop Clarification for High-Ambiguity Daily TasksStep-GUI Technical Report🟠Medium
- Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RLTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🟠Medium
- Closed-Loop Autoregressive Training to Mitigate Prerouter Feature DriftThe Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction🟡Fairly easy