Dense Process Reward Optimization for Long-Horizon GUI Agents
Provide dense, step-by-step feedback to long-horizon GUI agents by training a dedicated process reward model on step-level critic data.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 3.1.2, the authors describe using Trajectory-aware Relative Policy Optimization (TRPO) where the advantage estimate from a single terminal reward R(tau) is 'uniformly distributed to every action taken within that trajectory'.
“This approach circumvents the challenge of assigning per-step rewards, a task that is nearly impossible to perform accurately in complex GUI interactions.”
Limitation
Uniform credit assignment is inefficient for long-horizon tasks (e.g., 20+ steps). If an agent performs 19 correct steps and fails on the 20th, TRPO penalizes all previous correct actions equally, slowing down convergence and failing to reinforce partial successes.
Proposed Solution
Train a dedicated Process Reward Model (PRM) using the data already generated by the 'Step-Level Critic' (described in Section 6.3). Integrate this PRM into the RL loop to provide dense, step-wise reward signals combined with the sparse terminal reward. This allows the policy to distinguish between a 'bad plan' and a 'good plan with one bad execution step'.
Why This Matters
Improving credit assignment is critical for scaling GUI agents to complex, real-world workflows where tasks may span dozens of screens and actions.
How to Get Started
How to test it: Hypothesis: Integrating a PRM will improve long-horizon task completion over sparse rewards. Baseline: The current sparse-reward TRPO policy model. Metric: Success rate on OSWorld-Verified. Stop if success rate improvement is less than 1.5% absolute.
- Extract step-level critic trajectories and intermediate annotations from existing training logs.
- Construct a dataset of positive and negative action prefixes to train a binary classification process reward model.
- Integrate the trained process reward model into the policy optimization loop to combine step-wise and terminal rewards.
- Benchmark the updated agent on complex GUI tasks against a baseline using only terminal rewards.
Code linked to this paper on Hugging Face: github.com/X-PLUG/MobileAgent
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like 'GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks' and 'PRO-CUA: Process-Reward Optimization for Computer Use Agents' explore process reward modeling and optimization for GUI tasks. However, these specific references do not detail training a dedicated PRM using data generated by a specific 'Step-Level Critic' configuration as proposed. The listed works provide context on step-level filtering and verification but leave the exact architectural integration open.
- GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks · 2025 · 34 citations
- GUI-PRA: Process Reward Agents for GUI Tasks · 2026 · 13 citations
- WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering · 2026 · 13 citations
- PRO-CUA: Process-Reward Optimization for Computer Use Agents · 2026 · 1 citations
We searched OpenAlex for “Provide dense, step-by-step feedback to long-horizon GUI agents by training a dedicated process reward model on step-level critic data.” and “Dense Process Reward Optimization for Long-Horizon GUI Agents” and an AI compared the idea with the top 9 results. A small search can miss relevant work.
Source Paper
Mobile-Agent-v3: Foundamental Agents for GUI AutomationJiabo Ye, Xi Zhang, Haiyang Xu +12 more · 2025
This paper presents GUI-Owl and Mobile-Agent-v3, a foundational end-to-end multimodal model and multi-agent framework designed for cross-platform GUI automation (Mobile, PC, Web). The authors introduce a self-evolving data production pipeline that synthesizes large-scale training data through model rollouts, critic-based filtering, and VLM-generated guidance, followed by a scalable reinforcement learning framework using Trajectory-aware Relative Policy Optimization (TRPO) to handle long-horizon tasks.
Related Ideas
- Attention-Guided Temporal Credit Assignment for Segment-Based Online Reinforcement LearningGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟠Medium
- Granular Credit Assignment via Learned Value FunctionsAgent Lightning: Train ANY AI Agents with Reinforcement Learning🟠Medium
- Reinforcement Learning for Long-Horizon GUI Agent NavigationPenguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders⛔Very hard
- Safety-Tuned Process Reward Models for Infeasible Task RefusalEnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings🟠Medium
- Efficiency-Weighted Reward Modeling for GRPOMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium