Reinforcement Learning from Execution Failures (RLEF)
Incorporate execution failure trajectories into training with penalized rewards and self-correction loops to teach agents how to recover from runtime errors.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.3 states that rollouts encountering timeouts, runtime deaths, or execution errors are filtered out ('Mark as broken trajectory') and excluded from the training set used for policy optimization.
“During rollout generation, the agent may produce invalid or non-executable Python code due to timeouts, runtime failures, or invalid multimodal outputs, e.g., exceeding image limits or failing to render any image.”
Limitation
By filtering out broken trajectories, the model is denied the learning signal required to avoid invalid syntax or logic errors. It relies solely on the SFT initialization for correctness, making it brittle to out-of-distribution coding tasks.
Proposed Solution
Modify the training loop to retain broken trajectories but assign them a penalized reward (e.g., -1). Furthermore, implement 'Self-Correction RL' where the execution error message is appended to the context, and the agent is rewarded for recovering from the error within the same episode.
Why This Matters
This would significantly increase the robustness of the agent, allowing it to self-heal during deployment rather than crashing when it generates slightly malformed code.
How to Get Started
How to test it: We hypothesize that assigning penalized rewards to broken trajectories and feeding back execution error messages will improve self-correction and out-of-distribution robustness. We will baseline against standard rollout filtering, measuring code execution success rate on TIR-Bench, and stop if error recovery rate does not improve by 5%.
- Modify the data collection pipeline to retain broken or timed-out trajectories instead of discarding them.
- Assign negative reward penalties to failed trajectories within the reinforcement learning dataset.
- Extend the environment to append error messages back into the agent context for subsequent steps.
- Train the policy on error-recovery episodes and measure the increase in overall task completion rates.
Code linked to this paper on Hugging Face: github.com/agents-x-project/PyVision-RL
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed literature consists of surveys, frameworks, and diagnostic tools such as From Anomalies to Failures: Constructing Causal Error Graphs for Agentic Trace Diagnosis and Agent World Model. None of these papers cover the specific reinforcement learning loop modification involving penalized execution failure retention and self-correction error-message appending.
We searched OpenAlex for “Incorporate execution failure trajectories into training with penalized rewards and self-correction loops to teach agents how to recover from runtime errors.” and “Reinforcement Learning from Execution Failures (RLEF)” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
PyVision-RL: Forging Open Agentic Vision Models via RLShitian Zhao, Shaoheng Lin, Ming Li +4 more · 2026
This paper introduces PyVision-RL, a framework for training open-weight multimodal agents (based on Qwen2.5-VL) to effectively use Python tools for image and video understanding. To address 'interaction collapse'—where models stop using tools after RL fine-tuning—the authors propose an oversampling-filtering-ranking training strategy and an accumulative tool reward mechanism.
Related Ideas
- Off-Policy Correction for Asynchronous Agent RLGLM-5: from Vibe Coding to Agentic Engineeringđźź Medium
- Self-Correction via Test-Driven Reinforcement LearningOpenDevin: An Open Platform for AI Software Developers as Generalist Agentsâ›”Very hard
- Sample-Efficient Online Policy Optimization for Rapid Agent AdaptationMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildđźź Medium
- Centralized Training with Decentralized Execution (CTDE) for Multi-Agent SystemsAgent Lightning: Train ANY AI Agents with Reinforcement Learningđź”´Hard
- Interactive Policy Refinement via World Model RLGigaBrain-0: A World Model-Powered Vision-Language-Action Modelâ›”Very hard