Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchRelated work found

Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness Edits

Cross-check proposed agent edits with specification-derived adversarial probes and multi-seed reruns to prevent shared model assumptions from passing self-written verification scripts.

Time

several weeks

Compute level

Single GPU or less

Resources

Workstation with Docker container runtime, API access to Claude Opus 4.8 and Gemini 3.5 Flash/GLM-5, and WorkBuddy-Bench plus LAB.

What the paper used

Frozen API executors (Gemini 3.5 Flash, GLM-5) and editor/selector (Claude Opus 4.8 at high reasoning effort) run inside OpenCode/Codex Docker containers (200-turn and 3,600s rollout caps), spending roughly 1-100M tokens per task across WorkBuddy-Bench (100 test tasks), JobBench (34 test tasks), and LAB (100 test tasks) over 3 runs.

Prerequisites

Software testing, LLM agent self-correction, and statistical variance reduction.

Scope

modify existing code

The Gap

In Section 2.2.1, Appendix E.6 (Figure 14), and Appendix F, candidate harnesses are tested via a single probe rollout per candidate, and behavioral contrasts plus self-written verification scripts (e.g., validator cache and header injection cases on WB-code) guide composition without external ground truth.

“These signals help assess the effects of a proposed change, but remain sensitive to execution variability and shared model biases.”
From the paper, Appendix F, page 29. Matched against the PDF text.

Limitation

Single-rollout probes conflate execution stochasticity with genuine edit effects, and when an editor introduces a faulty domain assumption or writes a matching self-checker script, the flawed premise propagates across composed candidates and passes self-checks while failing hidden tests.

Proposed Solution

Introduce decoupled falsification probing into the Probe and Compose stages: (1) generate independent specification-derived edge-case micro-tests (e.g., case-sensitivity, structural equivalence, and boundary inputs extracted solely from task.md and source schemas without seeing the candidate's scripts) to stress-test candidate code/scripts; (2) when an editor proposes a hard prescriptive guardrail or domain rule that contradicts any base rollout's behavior, require a paired counterfactual probe (running with vs. without the restrictive rule on matched seeds) before composition; and (3) allocate targeted multi-seed replication (e.g., 2-3 seeds) specifically to edits showing high behavioral variance rather than uniform single-rollout probing.

Why This Matters

Decoupling verification from editor assumptions and controlling for rollout variance should lower regression rates caused by restrictive priors and unverified self-checkers.

How to Get Started

How to test it: Test whether specification-derived falsification tests and targeted counterfactual re-probes reduce negative-delta regressions (tasks scoring below base harness) on WorkBuddy-Bench Code and LAB without exceeding the k=5 token budget. Compare regression frequency and mean score against standard parallel k=5 Harness^2; stop if overhead from extra probes outweighs regression reductions.

  1. Identify all tasks across WorkBuddy-Bench and LAB where Harness^2 underperforms the base harness and categorize whether failures stem from shared restrictive rules or rollout variance.
  2. Implement a separate specification-tester prompt that reads only task.md and source schemas to output executable edge-case assertions for scripts and deliverables.
  3. Modify the Compose stage to flag restrictive rules that forbid behaviors seen in base rollouts and trigger a single matched counterfactual check before merging.
  4. Evaluate regression rate and net benchmark score on WorkBuddy-Bench and LAB under matched total token budgets.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

While [0] and [8] discuss adversarial correctness and verified detection of anomalies in agents, and [9] provides foundational work on natural-language harnesses, the specific mechanism of using decoupled specification-derived micro-tests to perform counterfactual verification of harness edits is novel relative to these results.

We searched OpenAlex and arXiv for “verifier-free agent self-improvement adversarial probing shared model bias” and “Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness Edits” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Harness^2: Recursive Agent Harnessing for an Open World

Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu +6 more · 2026

This paper introduces Harness^2, a plug-and-play recursive agent harnessing framework that enables LLM agents to optimize their execution harnesses at test time in open-world settings without ground-truth verifiers. It couples task-level contrastive refinement (Propose-Probe-Compose) with harnessing-level recursion that distills reusable harness components and lessons into an evolving improvement library across tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn