Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Context-Aware Mixture-of-Adapters for Unified Agents

Preserve specialist capabilities in unified models without destructive merging conflicts by routing tasks through a mixture of modular adapters.

Time

1-2 weeks

Compute level

Multiple GPUs

Resources

Multi-GPU training nodes (e.g., 8x A100/H100) to train adapter modules and the routing network over fine-tuned base models.

What the paper used

Two dense variants (2B and 8B) and one mixture-of-experts variant (30B-A3B) trained on 10 billion tokens across 30+ datasets

Prerequisites

Transformer architectures, PEFT (LoRA)

Scope

Modify existing code

The Gap

Section 2.4 describes merging specialized checkpoints (Grounding, Web, Mobile) into a single model using TIES-Merge. While better than linear merging, TIES-Merge still incurs performance trade-offs (e.g., ScreenSpot-Pro drops 1.4% compared to the specialist model).

“A major distinction from UI-Venus-1.0 is that UI-Venus-1.5 is a purely end-to-end model, which greatly simplifies deployment for users. To achieve this, we first conduct finetuning for grounding, web, and mobile domains by using domain-specific reward functions, data, and prompts. Once we obtain these three specialized models, we apply a model merge strategy.”
From the paper, Section 1, page 3. Matched against the PDF text.

Limitation

Static weight merging forces a 'one-size-fits-all' parameter set, causing interference between distinct domain skills (e.g., pixel-precise grounding vs. semantic web navigation) and preventing the unified model from matching the peak performance of specialized experts.

Proposed Solution

Replace destructive weight merging with a Mixture-of-Adapters (MoA) approach. Retain the specialized fine-tuned weights as modular LoRA adapters and train a lightweight gating network (router) to dynamically activate the relevant adapter at inference time based on the visual context (OS detection) or task type.

Why This Matters

This allows a single agent to maintain 'Specialist-level' performance across all domains without the degradation caused by parameter interference, offering a strictly superior unified model.

How to Get Started

How to test it: Hypothesis: Mixture-of-Adapters (MoA) with a dynamic router prevents inter-domain interference better than static weight merging. Baseline: UI-Venus-1.5 using TIES-Merge across specialized domain checkpoints. Metric: Average accuracy across specialized benchmarks (ScreenSpot-Pro, WebVoyager, AndroidWorld). Stopping condition: If MoA routing latency exceeds 50ms per step or if multi-domain accuracy drops below the TIES-Merge baseline.

  1. Extract specialized fine-tuned checkpoints for grounding, web, and mobile tasks as separate modular LoRA adapters.
  2. Design a lightweight gating network to classify incoming visual contexts and tasks at inference time.
  3. Train the router network on validation sets and measure performance retention against individual specialist models.

Code linked to this paper on Hugging Face: github.com/inclusionAI/UI-Venus

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters and CuMA: Aligning LLMs with Sparse Cultural Values via Demographic-Aware Mixture of Adapters explore combining adapters with routing mechanisms. However, they do not specifically address context-aware mixture-of-adapters for unified agents based on visual OS detection as proposed. Thus, the specific application remains partially explored.

We searched OpenAlex for “Preserve specialist capabilities in unified models without destructive merging conflicts by routing tasks through a mixture of modular adapters.” and “Context-Aware Mixture-of-Adapters for Unified Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

UI-Venus-1.5 Technical Report

Veuns-Team, Changlong Gao, Zhangxuan Gu +24 more · 2026

This paper introduces UI-Venus-1.5, an end-to-end multimodal GUI agent designed to navigate mobile and web environments via a unified model. The system is trained through a three-stage pipeline: large-scale Mid-Training on 10B tokens of GUI data, task-specific Offline Reinforcement Learning (RL), and massive Online RL on a Device-as-a-Service infrastructure, culminating in a model merging strategy that unifies specialized capabilities.

Related Ideas

Is this a good idea?

Share onXLinkedIn