Evaluation on Repository-Scale Benchmarks (SWE-bench)
Extend the evaluation harness of autonomous development agents to support repository-scale benchmarks so systems can be tested on real GitHub issues and pull requests.
Source paper
Michele Tufano, Anisha Agarwal, Jinu Jang, Roshanak Zilouchian Moghaddam, Neel Sundaresan
Generated by AI. The paper’s authors did not write these.
Extend the evaluation harness of autonomous development agents to support repository-scale benchmarks so systems can be tested on real GitHub issues and pull requests.
Introduce a memory summarizer module into the agent conversation manager to replace verbose raw build logs with concise semantic summaries while retaining recent details.
Integrate grammar-constrained decoding into tool command generation so language models strictly follow defined schemas and eliminate syntax-related parsing failures.
AutoDev is a fully automated AI-driven software development framework that enables LLM-based agents to autonomously plan and execute complex software engineering tasks. Unlike standard coding assistants that only suggest snippets, AutoDev agents operate within a secure Docker environment to edit files, retrieve context, build code, run tests, and execute git commands iteratively until objectives are met.
The framework orchestrates agents using a Conversation Manager that parses natural language commands into executable actions (File Editing, Build, Test, Git) provided by a Tools Library. An Agent Scheduler manages multi-agent collaboration (e.g., Round Robin), while an Evaluation Environment executes commands in isolated Docker containers to ensure security and capture feedback (logs, errors) for the agents.