Evaluation on Repository-Scale Benchmarks (SWE-bench)
Extend the evaluation harness of autonomous development agents to support repository-scale benchmarks so systems can be tested on real GitHub issues and pull requests.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The paper evaluates AutoDev primarily on HumanEval (Section 3 and 6.2), a dataset of small, self-contained algorithmic problems. Section 6.2 acknowledges that current metrics fall short in capturing essential programming aspects of real-world codebases.
“While our current evaluation relies on HumanEval for assessing the effectiveness of AutoDev in two coding tasks, our future work aims to extend this assessment to more challenging and real-world datasets, such as those offered by the Copilot Evaluation Harness.”
Limitation
Evaluating on isolated functions fails to test the system's ability to navigate complex dependency graphs, handle large file contexts, or modify legacy code in existing repositories, which are the primary challenges in real-world software engineering.
Proposed Solution
Extend the AutoDev evaluation harness to support SWE-bench, a benchmark consisting of real GitHub issues and Pull Requests. This involves adapting the Docker environment to dynamically install repository-specific dependencies and modifying the retrieval tools to handle codebase-wide context searching.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Validates the framework's practical utility for actual software maintenance tasks rather than just algorithmic puzzles, potentially bridging the gap to commercial adoption.
How to Get Started
How to test it: The hypothesis is that integrating SWE-bench will measure repository-level navigation capabilities. The baseline is current function-level HumanEval performance. The evaluation metric is pass rate on SWE-bench tasks. A passing result showing robustness would maintain at least 40% resolution rate.
- Clone the SWE-bench repository and review the evaluation harness and Docker environment specifications.
- Integrate the agent execution framework with the SWE-bench Docker container to handle dynamic dependency installation.
- Update the agent's codebase retrieval and search tools to support repository-wide context indexing and lookup.
- Run the evaluation pipeline on a subset of SWE-bench Lite issues to measure resolve rates and parsing failures.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? introduces the SWE-bench benchmark for evaluating language models on real GitHub issues and pull requests, requiring dynamic environment setup and retrieval strategies. However, the listed papers do not specifically detail the integration of this benchmark into the AutoDev harness as proposed.
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · 2023 · 2593 citations
We searched OpenAlex for “Extend the evaluation harness of autonomous development agents to support repository-scale benchmarks so systems can be tested on real GitHub issues and pull requests.” and “Evaluation on Repository-Scale Benchmarks (SWE-bench)” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
AutoDev: Automated AI-Driven DevelopmentMichele Tufano, Anisha Agarwal, Jinu Jang +2 more · 2024
AutoDev is a fully automated AI-driven software development framework that enables LLM-based agents to autonomously plan and execute complex software engineering tasks. Unlike standard coding assistants that only suggest snippets, AutoDev agents operate within a secure Docker environment to edit files, retrieve context, build code, run tests, and execute git commands iteratively until objectives are met.
Related Ideas
- Cross-Repository Micro-Skill Generalization via Project Manifest EmbeddingsTokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents🟡Fairly easy
- Hermetic Harness-Agent Environment Isolation Against Dependency-Mediated Answer LeakageSage S2.1 Flash Technical Report🟡Fairly easy
- Generalizing OpenSeeker: Integrating Heterogeneous Tool CapabilitiesOpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data🔴Hard
- Hierarchical Specification-to-Patch Synthesis for Large-Scale Repository ChangesSWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations🟠Medium
- Utility-Tracked Consolidation and Failure-Aware Retrieval for Harness Improvement LibrariesHarness^2: Recursive Agent Harnessing for an Open World🟡Fairly easy