Skip to content
AI-generated research idea🟢Easy🔧Minor TweakRelated work found

Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable Selection

Augment final output selection with trajectory-derived behavioral invariants and requirement-coverage checklists to prevent selectors from picking incomplete or non-conforming deliverables.

Time

a few days

Compute level

Single GPU or less

Resources

Workstation with Docker container runtime, API access to Claude Opus 4.8 and Gemini 3.5 Flash, and saved candidate pools or re-runs on WorkBuddy-Bench and LAB.

What the paper used

Frozen API executors (Gemini 3.5 Flash, GLM-5) and editor/selector (Claude Opus 4.8 at high reasoning effort) run inside OpenCode/Codex Docker containers (200-turn and 3,600s rollout caps), spending roughly 1-100M tokens per task across WorkBuddy-Bench (100 test tasks), JobBench (34 test tasks), and LAB (100 test tasks) over 3 runs.

Prerequisites

LLM-as-a-judge design, agent trajectory analysis, and structured evaluation.

Scope

modify existing code

The Gap

In Section 2.2.2 (Prompt 5) and Appendix E.6 (Figure 14), when k > 1, a terminal selector (Claude Opus 4.8) chooses among anonymized final deliverables solely by inspecting context/task.md, context/source/, and rendered artifacts, completely blinded to execution trajectories and harness metadata.

Limitation

Blindly judging only rendered deliverable text causes severe score regressions (such as dropping from 96 to 55 on WB-code, 100 to 70 on WB-web, and 54 to 26 on LAB) because the selector over-weights surface polish or precision over requirement coverage and misses execution-time failures like unexposed runtime state or failing checks.

Proposed Solution

Upgrade Prompt 5 into a two-phase structured selection pipeline: (1) decompose context/task.md into an explicit atomic requirement-coverage matrix (listing every requested entity, file, state transition, and format rule) before viewing candidates, scoring each candidate strictly on requirement coverage alongside factual precision; and (2) attach sanitized, harness-agnostic execution summaries (e.g., whether runtime/test commands exited cleanly, browser DOM/console check results, and tool error counts, while still masking harness diffs and candidate provenance) so the selector can filter out deliverables produced by broken executions without introducing harness-complexity bias.

Why This Matters

Reducing blind-selection regressions could recover substantial performance already achieved by candidate harnesses in the pool without requiring additional generation rollouts.

How to Get Started

How to test it: Test whether requirement-coverage decomposition and sanitized execution-summary signals reduce selection regret (difference between oracle best-of-pool score and selected score) on WorkBuddy-Bench and LAB at k=5. Compare against Prompt 5 on selection regret and overall benchmark score; stop if selection regret does not decrease or precision-sensitive tasks regress.

  1. Collect the k=5 candidate deliverable pools and ground-truth retrospective scores from WorkBuddy-Bench (Code, Web) and LAB runs using github.com/google-research/harness2.
  2. Measure baseline selection regret (oracle pool max minus Prompt 5 pick) to isolate selection failures from proposal failures.
  3. Implement a pre-selection task-decomposition step generating an explicit requirement checklist and append sanitized tool/exit-code summaries to candidate views.
  4. Re-run selection offline across stored pools first, then integrate into online k=5 runs on WorkBuddy-Bench and LAB.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Existing research in [7] and [9] touches on trace-aware reasoning and evidence tracking for agents, while [0] and [4] discuss LLM-based evaluation metrics. The proposal to combine a structured requirement-coverage matrix with harness-agnostic execution summaries specifically to minimize harness-complexity bias is not explicitly detailed in the listed works.

We searched OpenAlex and arXiv for “LLM judge blind candidate selection requirement coverage execution trace” and “Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable Selection” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Harness^2: Recursive Agent Harnessing for an Open World

Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu +6 more · 2026

This paper introduces Harness^2, a plug-and-play recursive agent harnessing framework that enables LLM agents to optimize their execution harnesses at test time in open-world settings without ground-truth verifiers. It couples task-level contrastive refinement (Propose-Probe-Compose) with harnessing-level recursion that distills reusable harness components and lessons into an evolving improvement library across tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn