Context-Aware Mixture-of-Adapters for Unified Agents
Preserve specialist capabilities in unified models without destructive merging conflicts by routing tasks through a mixture of modular adapters.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 2.4 describes merging specialized checkpoints (Grounding, Web, Mobile) into a single model using TIES-Merge. While better than linear merging, TIES-Merge still incurs performance trade-offs (e.g., ScreenSpot-Pro drops 1.4% compared to the specialist model).
“A major distinction from UI-Venus-1.0 is that UI-Venus-1.5 is a purely end-to-end model, which greatly simplifies deployment for users. To achieve this, we first conduct finetuning for grounding, web, and mobile domains by using domain-specific reward functions, data, and prompts. Once we obtain these three specialized models, we apply a model merge strategy.”
Limitation
Static weight merging forces a 'one-size-fits-all' parameter set, causing interference between distinct domain skills (e.g., pixel-precise grounding vs. semantic web navigation) and preventing the unified model from matching the peak performance of specialized experts.
Proposed Solution
Replace destructive weight merging with a Mixture-of-Adapters (MoA) approach. Retain the specialized fine-tuned weights as modular LoRA adapters and train a lightweight gating network (router) to dynamically activate the relevant adapter at inference time based on the visual context (OS detection) or task type.
Why This Matters
This allows a single agent to maintain 'Specialist-level' performance across all domains without the degradation caused by parameter interference, offering a strictly superior unified model.
How to Get Started
How to test it: Hypothesis: Mixture-of-Adapters (MoA) with a dynamic router prevents inter-domain interference better than static weight merging. Baseline: UI-Venus-1.5 using TIES-Merge across specialized domain checkpoints. Metric: Average accuracy across specialized benchmarks (ScreenSpot-Pro, WebVoyager, AndroidWorld). Stopping condition: If MoA routing latency exceeds 50ms per step or if multi-domain accuracy drops below the TIES-Merge baseline.
- Extract specialized fine-tuned checkpoints for grounding, web, and mobile tasks as separate modular LoRA adapters.
- Design a lightweight gating network to classify incoming visual contexts and tasks at inference time.
- Train the router network on validation sets and measure performance retention against individual specialist models.
Code linked to this paper on Hugging Face: github.com/inclusionAI/UI-Venus
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters and CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapters explore combining adapters with routing mechanisms. However, they do not specifically address context-aware mixture-of-adapters for unified agents based on visual OS detection as proposed. Thus, the specific application remains partially explored.
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters · 2024 · 371 citations
- CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapters · 2026 · 4 citations
We searched OpenAlex for “Preserve specialist capabilities in unified models without destructive merging conflicts by routing tasks through a mixture of modular adapters.” and “Context-Aware Mixture-of-Adapters for Unified Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
UI-Venus-1.5 Technical ReportVeuns-Team, Changlong Gao, Zhangxuan Gu +24 more · 2026
This paper introduces UI-Venus-1.5, an end-to-end multimodal GUI agent designed to navigate mobile and web environments via a unified model. The system is trained through a three-stage pipeline: large-scale Mid-Training on 10B tokens of GUI data, task-specific Offline Reinforcement Learning (RL), and massive Online RL on a Device-as-a-Service infrastructure, culminating in a model merging strategy that unifies specialized capabilities.
Related Ideas
- Hierarchical Multi-Specialist Coordination Protocol for GUI-MCPStep-GUI Technical Report🟠Medium
- Sandboxed Evaluation of Agentic CapabilitiesGPT-4 Technical Report⛔Very hard
- Data Utility Feedback Loop for Agentic OrchestrationDataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI🔴Hard
- Centralized Training with Decentralized Execution (CTDE) for Multi-Agent SystemsAgent Lightning: Train ANY AI Agents with Reinforcement Learning🔴Hard
- Dynamic State Clustering for Group-Relative Policy Optimization in Asynchronous RLOpenClaw-RL: Train Any Agent Simply by Talking🟠Medium