Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RL
A method that applies token-level off-policy importance weighting to partial rollout trajectories to avoid discarding entire query groups during asynchronous multi-turn agent reinforcement learning.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 4.4 describes the asynchronous reinforcement learning workflow with partial rollout in TeleGym, where heterogeneous task durations cause agent trajectories to span multiple policy updates. To address policy lag, TeleGym employs a stale-data filtering heuristic that rejects an entire query group if any constituent trajectory is identified as old, in order to preserve the group membership required for group-relative reward normalization.
“Optional stale-data filtering uses observed policy versions to identify old trajectories and removes the entire query group when a constituent trajectory is rejected, preserving the membership used for group-relative reward processing. This is a trajectory-level heuristic rather than a strict token-wise age bound.”
Limitation
The stale-data filtering heuristic operates at a coarse trajectory level by discarding entire query groups whenever a single trajectory becomes stale, causing substantial training sample waste and discarding useful intermediate tokens collected during multi-turn interactions.
Proposed Solution
Implement a token-level or segment-level off-policy correction mechanism, such as truncated importance sampling or V-trace adapted for group-relative policy optimization (GRPO). Instead of dropping entire query groups when policy version drift occurs, compute the importance sampling ratio between the current policy and the sampling-time policy log-probabilities recorded by the server for each generated token segment. Adjust the group-relative baseline and advantage weighting using importance-corrected sample weights, allowing valid prefixes and completed steps from lagging trajectories to contribute gradient signals without corrupting optimization.
Why This Matters
This approach is expected to significantly increase sample efficiency in agent RL pipelines, reducing wasted compute on discarded rollouts and allowing long-horizon multi-turn tasks to train stably even under frequent policy updates.
How to Get Started
How to test it: Evaluate on multi-turn code and reasoning benchmarks (such as LiveCodeBench and GSM8K) under high simulated policy lag; the hypothesis is that token-level importance weighting achieves equal or superior sample efficiency while utilizing at least 30% more generated rollout tokens than trajectory-level group dropping, stopping if importance weight variance destabilizes policy convergence.
- Set up an asynchronous RL pipeline using an open-source framework such as Slime or OpenRLHF configured with group-relative policy optimization.
- Implement artificial policy publication delays to reproduce TeleGym's trajectory-level group dropping behavior as an evaluation baseline.
- Add token-wise importance ratio calculation and clipping to the loss function using recorded behavior-policy log probabilities.
- Benchmark gradient variance, sample retention rate, and task performance across varied rollout latency distributions.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as 'COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning' and 'Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization' directly address token-level or coupled off-policy corrections for asynchronous LLM reinforcement learning. These works explore similar mechanisms to handle stale or lagging trajectories and logits during training.
- COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning · 2026 · 1 citations
- Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction · 2026 · 6 citations
- Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization · 2026 · 1 citations
We searched OpenAlex and arXiv for “asynchronous reinforcement learning agent off-policy importance weighting” and “Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RL” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
TeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and FrameworksXingchen AGI Lab, Baotian Hu, Bingkai Yang +136 more · 2026
This technical report presents Xing 4, a family of large language models scaling up to hundreds of billions of parameters trained natively on Ascend 910C NPUs using MindSpore HyperParallel. The report establishes design principles combining sparse attention, wider-and-shallower MoE topologies, and speculative decoding, supported by systems optimizations including hybrid sharded data parallelism, owner-based Muon execution, and CPU core isolation. It also introduces TeleGym, a unified execution and reinforcement learning platform for long-horizon agent and reasoning workloads.
Related Ideas
- Off-Policy Correction for Asynchronous Agent RLGLM-5: from Vibe Coding to Agentic Engineering🟠Medium
- Dynamic State Clustering for Group-Relative Policy Optimization in Asynchronous RLOpenClaw-RL: Train Any Agent Simply by Talking🟠Medium
- Sample-Efficient Online Policy Optimization for Rapid Agent AdaptationMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild🟠Medium
- Attention-Guided Temporal Credit Assignment for Segment-Based Online Reinforcement LearningGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟠Medium
- Efficiency-Weighted Reward Modeling for GRPOMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium