Skip to content
AI-generated research idea🟢Easy🔧Minor TweakNo close match found

Reinforcement Learning from Execution Failures (RLEF)

Incorporate execution failure trajectories into training with penalized rewards and self-correction loops to teach agents how to recover from runtime errors.

Time

12 hours

Compute level

Large cluster

Resources

8 NVIDIA H100 GPUs for 700 RL steps.

What the paper used

Trained for 700 RL steps with oversampling batch size 32, training batch size 16, group size 8, and learning rate 1 x 10^-6 on 8 H100 GPUs.

Prerequisites

basic ML knowledge, Python execution sandboxing

Scope

modify existing code

The Gap

Section 3.3 states that rollouts encountering timeouts, runtime deaths, or execution errors are filtered out ('Mark as broken trajectory') and excluded from the training set used for policy optimization.

“During rollout generation, the agent may produce invalid or non-executable Python code due to timeouts, runtime failures, or invalid multimodal outputs, e.g., exceeding image limits or failing to render any image.”
From the paper, Section 3.3, page 4. Matched against the PDF text.

Limitation

By filtering out broken trajectories, the model is denied the learning signal required to avoid invalid syntax or logic errors. It relies solely on the SFT initialization for correctness, making it brittle to out-of-distribution coding tasks.

Proposed Solution

Modify the training loop to retain broken trajectories but assign them a penalized reward (e.g., -1). Furthermore, implement 'Self-Correction RL' where the execution error message is appended to the context, and the agent is rewarded for recovering from the error within the same episode.

Why This Matters

This would significantly increase the robustness of the agent, allowing it to self-heal during deployment rather than crashing when it generates slightly malformed code.

How to Get Started

How to test it: We hypothesize that assigning penalized rewards to broken trajectories and feeding back execution error messages will improve self-correction and out-of-distribution robustness. We will baseline against standard rollout filtering, measuring code execution success rate on TIR-Bench, and stop if error recovery rate does not improve by 5%.

  1. Modify the data collection pipeline to retain broken or timed-out trajectories instead of discarding them.
  2. Assign negative reward penalties to failed trajectories within the reinforcement learning dataset.
  3. Extend the environment to append error messages back into the agent context for subsequent steps.
  4. Train the policy on error-recovery episodes and measure the increase in overall task completion rates.

Code linked to this paper on Hugging Face: github.com/agents-x-project/PyVision-RL

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

The listed literature consists of surveys, frameworks, and diagnostic tools such as From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis and Agent World Model. None of these papers cover the specific reinforcement learning loop modification involving penalized execution failure retention and self-correction error-message appending.

We searched OpenAlex for “Incorporate execution failure trajectories into training with penalized rewards and self-correction loops to teach agents how to recover from runtime errors.” and “Reinforcement Learning from Execution Failures (RLEF)” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

PyVision-RL: Forging Open Agentic Vision Models via RL

Shitian Zhao, Shaoheng Lin, Ming Li +4 more · 2026

This paper introduces PyVision-RL, a framework for training open-weight multimodal agents (based on Qwen2.5-VL) to effectively use Python tools for image and video understanding. To address 'interaction collapse'—where models stop using tools after RL fine-tuning—the authors propose an oversampling-filtering-ranking training strategy and an accumulative tool reward mechanism.

Related Ideas

Is this a good idea?

Share onXLinkedIn