Dynamic Working-Memory Pruning for Search Agents to Overcome Reinforcement Learning Saturation
Prune redundant retrieved text dynamically during search agent rollouts to prevent context saturation and reward plateaus during long-horizon reinforcement learning.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 6.3.1 and Figure 41, the authors analyze the training trajectory of the 375B-A23B search expert under GRPO. The search expert improves in base reward initially (55.4% to 61.8%) but flattens completely after rollout step 20, with trajectory length reaching 62.0k tokens.
“This expert is in fact context-bound: its longest trajectory sits at 97.6k tokens at every rollout step (97,608–97,712 across all 40) against a 96k window and truncation rises 14.3% → 22.4%, so the exported checkpoint sits on a plateau rather than on a rise.”
Limitation
Search agent trajectories become context-bound because approximately 84% of the trajectory consists of raw retrieved context rather than reasoning, causing truncation to rise from 14.3% to 22.4% and plateauing performance gains.
Proposed Solution
Replace static raw document ingestion with an integrated dynamic context pruner and memory compression step. During rollout execution, after each search or fetch tool call, apply an attention-saliency or extractive summarization step that retains only query-relevant facts and extracts before appending to the running context. Incorporate an explicit context frugality penalty or efficiency reward into the GRPO objective to train the agent to discard irrelevant retrieved passages prior to next-step planning.
Why This Matters
Preventing context saturation would allow search agents to continue exploring complex, multi-hop queries over dozens of turns without hitting hard context limits or suffering from off-policy truncation bias.
How to Get Started
How to test it: Train the K2-Horizon-7B search expert using the proposed dynamic pruning against the baseline RL recipe from Table 29; the method succeeds if the average truncation rate stays below 5% and the final test accuracy on BrowseComp or complex multi-hop search tasks improves by at least 3 percentage points.
- Clone the public RL360 repository and configure the search expert environment with the local Wikipedia FAISS index.
- Instrument the search harness to log per-turn context accumulation and replicate the truncation plateau on a 7B model.
- Implement a post-retrieval extractive pruning tool that compresses fetched documents to under 1k tokens based on cross-attention scores.
- Run comparative GRPO training runs with and without dynamic pruning across 50 optimization steps.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning and SWE-Pruner explore context pruning and reinforcement learning for agents, but they do not specifically integrate attention-saliency working-memory pruning with efficiency rewards under GRPO for search agents.
- SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents · 2026 · 77 citations
- Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning · 2026 · 13 citations
We searched OpenAlex and arXiv for “reinforcement learning search agent context pruning truncation” and “Dynamic Working-Memory Pruning for Search Agents to Overcome Reinforcement Learning Saturation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
K2-Horizon: A Frontier Fleet of Open Source Language ModelsK2 Team, Zhengzhong Liu, Haonan Li +92 more · 2026
This technical report presents K2-Horizon, a family of six fully open language models ranging from 0.9B dense to a 375B mixture-of-experts (MoE) flagship activating 23B parameters, optimized for agentic and reasoning tasks. The authors open-source the complete development artifacts, including intermediate checkpoints, pre-training corpora with 10T synthetic tokens, the xLLM training framework, and the RL360 reinforcement learning stack. Key architectural innovations include Grouped RMSNorm, Mixture-of-Value Attention (MoVA), softplus attention gating, and Uno discrete diffusion adapters for accelerated decoding.
Related Ideas
- Dynamic Multi-Step Retrieval for Long-Horizon TasksRemember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution🟠Medium
- Importance-Weighted Semantic Memory Compaction for Ultra-Long Horizon TasksGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟡Fairly easy
- Hierarchical Retrieval-Augmented Self-Improvement Memory for Extended Optimization HorizonsSelf-Improvement Memory Enables Continual Recursive Self-Improvement🟡Fairly easy
- Compressive Memory for Agentic ContextsGLM-5: from Vibe Coding to Agentic Engineering🟠Medium
- Efficiency-Weighted Reward Modeling for GRPOMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium