Self-Correction via Test-Driven Reinforcement Learning
Use execution results from Docker test suites as a reinforcement learning reward signal to train a critic model that blocks failing code actions.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section A mentions 'Automatic workflow generation' and the limitation that current workflows are 'hand-crafted.' The current agents operate on a fixed Prompt-Action-Observation loop without learning from previous failures within the specific repository context.
Limitation
Agents repeat the same mistakes (e.g., syntax errors or incorrect library usage) because they lack a mechanism to update their policy based on the execution feedback (success/failure of unit tests) over time.
Proposed Solution
Implement a Reinforcement Learning (RL) layer using the Docker sandbox execution results as a reward signal. Specifically, use the outcome of `CmdRunAction` (running test suites) to provide a sparse reward. Train a 'Critic' model that intercepts the agent's proposed action and predicts the likelihood of test failure, rejecting actions that statistically lead to broken builds based on historical interaction data.
Why This Matters
Moves the system from a static executor to an adaptive learner that improves its coding accuracy the longer it interacts with a codebase.
How to Get Started
- Set up the Docker sandbox environment to reliably capture exit codes and test failure logs from `CmdRunAction`.
- Collect historical interaction trajectories to form a preference dataset of successful versus broken builds.
- Train a critic model to predict test execution outcomes based on proposed agent actions.
- Implement a rejection-sampling reinforcement learning loop that filters out actions flagged by the critic.
Code linked to this paper on Hugging Face: github.com/opendevin/opendevin
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as SPROUT and Code-A1 explore reinforcement learning for software engineering and code generation agents using test outcomes. However, the specific proposal of using a Critic model to intercept actions and predict test failure based on sandbox execution results leaves room for this concrete architecture.
- Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning · 2026 · 12 citations
- SPROUT: Scaling Agentic Reinforcement Learning for Long-Horizon Software Engineering · 2026 · 1 citations
We searched OpenAlex for “Use execution results from Docker test suites as a reinforcement learning reward signal to train a critic model that blocks failing code actions.” and “Self-Correction via Test-Driven Reinforcement Learning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
OpenDevin: An Open Platform for AI Software Developers as Generalist AgentsXingyao Wang, Boxuan Li, Yufan Song +21 more · 2024
OpenHands (formerly OpenDevin) is an open-source platform designed to facilitate the development and evaluation of AI agents capable of performing complex software engineering tasks. The platform provides a standardized event-stream architecture, a secure Docker-based runtime for code execution and web browsing, and a suite of benchmarks (such as SWE-Bench and WebArena) to assess agent performance against real-world scenarios.
Related Ideas
- Evaluating Continual Agent Adaptation on Open-World Human-in-the-Loop WorkloadsMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild⛔Very hard
- Reinforcement Learning from Execution Failures (RLEF)PyVision-RL: Forging Open Agentic Vision Models via RL🟢Easy
- Sample-Efficient Online Policy Optimization for Rapid Agent AdaptationMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild🟠Medium
- Numerical Stability-Aware RL for Dry ExperimentsProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows🔴Hard
- Reinforcement Learning for Long-Horizon GUI Agent NavigationPenguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders⛔Very hard