Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

NFR-Aware Benchmarking for Agentic Coding

Create an evaluation framework paired with explicit non-functional requirements to benchmark coding agents on both functional correctness and code quality constraints.

Time

several weeks

Compute level

Single GPU or less

Resources

Standard GPU for local model inference or LLM API access; Existing coding benchmark dataset (e.g., SWE-bench)

Prerequisites

LLM evaluation methodologies, Software Engineering principles (NFRs), Static Analysis tools (e.g., Pylint, Bandit)

Scope

Dataset augmentation (injecting NFR constraints into context), Evaluation harness extension (integrating static analysis for NFR verification)

The Gap

Section 4.3 and Table 3 reveal a significant gap where non-functional requirements (NFRs) like security and performance are only present in 14.5% of context files, despite their criticality for production-grade software.

Limitation

Existing benchmarks like SWE-bench primarily measure functional task completion but fail to evaluate whether an agent adheres to quality guardrails or NFR constraints specified in a project's persistent context.

Proposed Solution

Create an evaluation framework where coding tasks are paired with agent context files containing explicit NFR constraints (e.g., 'Do not use wildcard imports' or 'Sanitize all SQL inputs'). Use a dual-scoring metric that evaluates both functional correctness and strict adherence to the provided NFR context.

Why This Matters

By penalizing agents for generating insecure or inefficient code that violates project-level instructions, researchers can shift the focus from 'making code work' to 'building code well.'

How to Get Started

  1. Select an existing coding benchmark such as SWE-bench as the evaluation foundation.
  2. Annotate a subset of tasks with explicit non-functional requirements like security rules and performance constraints.
  3. Develop a dual-scoring automated evaluation script combining functional test runners and static analysis tools.
  4. Run baseline coding agents through the new benchmark framework to measure adherence rates.

Code linked to this paper on Hugging Face: github.com/openai/agents.md

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like 'DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents' explore joint security and functionality benchmarking, and 'Quality assurance of LLM-generated code: Addressing non-functional quality characteristics' discusses non-functional attributes. However, they do not specifically evaluate coding tasks paired with agent context files containing explicit NFR constraints using a dual-scoring metric.

We searched OpenAlex and arXiv for “Create an evaluation framework paired with explicit non-functional requirements to benchmark coding agents on both functional correctness and code quality constraints.” and “NFR-Aware Benchmarking for Agentic Coding” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Agent READMEs: An Empirical Study of Context Files for Agentic Coding

Worawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa +8 more · 2025

This paper presents the first large-scale empirical study of agent context files (such as CLAUDE.md and AGENTS.md), analyzing 2,303 files from 1,925 repositories. The study characterizes these files as complex, 'configuration-as-code' artifacts that are frequently maintained but lack critical non-functional requirements like security and performance.

Related Ideas

Is this a good idea?

Share onXLinkedIn