Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Efficiency-Weighted Reward Modeling for GRPO

Penalize redundant tool calls in reinforcement learning by incorporating a dynamic efficiency reward based on observation embedding similarity.

Time

several weeks

Compute level

Large cluster

Resources

Large compute cluster (for RL training)

Prerequisites

reinforcement learning

Scope

modify existing code

The Gap

In Section 5.3, the Reinforcement Learning reward function is defined as R(x, H) = alpha_c * R_correct - alpha_f * R_format. Section 6.4 subsequently admits that this leads to 'redundant retries' and 'marginal contributions' from excessive tool calls, as the model learns to spam interactions to maximize correctness without regard for computational cost or efficiency.

Limitation

The current RL reward structure incentivizes 'interactive scaling' (more steps) but fails to incentivize efficiency, leading to bloated trajectories where the agent performs redundant actions just to verify low-value information.

Proposed Solution

Modify the GRPO reward function to include a dynamic efficiency penalty. Introduce a term '- alpha_e * R_cost', where R_cost scales non-linearly with the number of tool calls that result in high semantic similarity to previous observations (measured via embedding cosine similarity). This forces the model to learn that 'interactive scaling' is only rewarded when new, distinct information is acquired, pruning redundant branches during training.

Why This Matters

This would significantly reduce inference latency and token costs, making the agent viable for real-world deployment where 600-turn trajectories are prohibitively expensive.

How to Get Started

  1. Implement a cosine similarity metric between consecutive tool outputs using a lightweight embedding model.
  2. Modify the GRPO reward function to include a non-linear cost penalty scaled by the semantic similarity score.
  3. Set up the RL training loop using standard open-source agent frameworks and codebases.
  4. Train the agent on multi-step reasoning benchmarks and evaluate tool call efficiency versus correctness.

Code linked to this paper on Hugging Face: github.com/MiroMindAI/MiroThinker

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting explores modifying GRPO reward functions using diversity-awareness, which is close to the idea of penalizing redundant or non-diverse tool calls. However, it does not explicitly use a dynamic efficiency penalty based on semantic similarity of tool outputs as proposed. None of the listed papers implement this specific penalty formulation.

We searched OpenAlex for “Penalize redundant tool calls in reinforcement learning by incorporating a dynamic efficiency reward based on observation embedding similarity.” and “Efficiency-Weighted Reward Modeling for GRPO” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling

MiroMind Team, Song Bai, Lidong Bing +51 more · 2025

MiroThinker v1.0 is an open-source research agent designed to push performance boundaries through 'interactive scaling,' where the model is trained to handle deeper and more frequent agent-environment interactions. By utilizing a 256K context window and a three-stage training pipeline involving SFT, DPO, and Reinforcement Learning (GRPO), the agent achieves state-of-the-art results on benchmarks like GAIA and BrowseComp, rivaling proprietary models like GPT-5-high.

Related Ideas

Is this a good idea?

Share onXLinkedIn