Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Utility-Tracked Consolidation and Failure-Aware Retrieval for Harness Improvement Libraries

Consolidate overlapping agent scaffolding components and rank past editing lessons by empirical utility so long-running self-improvement libraries stay compact and relevant.

Time

several weeks

Compute level

Single GPU or less

Resources

Workstation with Docker container runtime, API access to Claude Opus 4.8 and Gemini 3.5 Flash/GLM-5 (10^0-10^2M tokens across test streams), and the released Harness^2 codebase plus WorkBuddy-Bench, JobBench, and LAB.

What the paper used

Frozen API executors (Gemini 3.5 Flash, GLM-5) and editor/selector (Claude Opus 4.8 at high reasoning effort) run inside OpenCode/Codex Docker containers (200-turn and 3,600s rollout caps), spending roughly 1-100M tokens per task across WorkBuddy-Bench (100 test tasks), JobBench (34 test tasks), and LAB (100 test tasks) over 3 runs.

Prerequisites

LLM agent scaffolding, vector/hybrid retrieval, and automated software refactoring.

Scope

modify existing code

The Gap

In Section 2.3 and Appendix D.2, harnessing-level recursion appends each task's full adopted harness tree and an EXPERIENCE.md entry to a domain-specific library directory, while an index file lists all entries for the editor to inspect on demand during the Propose stage.

“As this library grows, future work could consolidate overlapping components, revise outdated experience, and refine retrieval according to the current task and available evidence.”
From the paper, Appendix G, page 29. Matched against the PDF text.

Limitation

As the deployment task stream grows, the append-only library accumulates redundant or overlapping skills, scripts, and outdated lessons without consolidation, pruning, or relevance-ranked retrieval, increasing context overhead and risking stale component reuse.

Proposed Solution

Implement an automated library curation module triggered periodically after task batches: (1) cluster adopted harness diffs and EXPERIENCE.md entries by behavioral failure signature using embedding similarity over trajectory summaries; (2) merge overlapping skills and Python helper scripts into canonical parameterized components with explicit trigger conditions; (3) track posterior utility scores for each library entry based on whether subsequent compositions retain or drop its reused edits in Experience Reflector logs, pruning or deprecating entries whose contrastive verdicts are consistently negative or superseded; and (4) replace full-index inspection with two-stage hybrid retrieval conditioned on the current reference trajectory's failure modes.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

If effective, this should keep the improvement library compact over long open-world streams while improving retrieval precision and reducing token overhead during proposal generation.

How to Get Started

How to test it: Test whether periodic consolidation and utility-ranked retrieval improves or preserves final task scores while shrinking library size and proposal tokens on an extended multi-domain stream (e.g., concatenating WorkBuddy-Bench domains and JobBench tasks). Compare against unmodified append-only Harness^2 on average score, library token footprint, and total tokens per task; stop if consolidated libraries underperform append-only baselines by more than 1 point at equal token budgets.

  1. Clone the released github.com/google-research/harness2 repository and reproduce the k=1 Gemini 3.5 Flash under OpenCode baseline on WorkBuddy-Bench and JobBench.
  2. Instrument the library directory to log entry count, file sizes, token usage of library reads, and Experience Reflector reuse/verdict rates across sequential tasks.
  3. Build a prototype consolidation script that groups EXPERIENCE.md entries by cue similarity and merges duplicate skills/scripts, testing it offline on completed WorkBuddy-Bench libraries.
  4. Integrate utility-weighted retrieval into the Propose stage prompt and evaluate on shuffled multi-benchmark task streams against the unmodified append-only library.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like [1] and [4] explore the general concept of self-evolving library management and taxonomy for agent skills, while [9] addresses the composition of harness-based intelligence. However, the specific combination of utility-tracked pruning based on posterior contrastive verdicts and failure-aware hybrid retrieval remains unaddressed by these sources.

We searched OpenAlex and arXiv for “continual agent skill library consolidation pruning retrieval” and “Utility-Tracked Consolidation and Failure-Aware Retrieval for Harness Improvement Libraries” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Harness^2: Recursive Agent Harnessing for an Open World

Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu +6 more · 2026

This paper introduces Harness^2, a plug-and-play recursive agent harnessing framework that enables LLM agents to optimize their execution harnesses at test time in open-world settings without ground-truth verifiers. It couples task-level contrastive refinement (Propose-Probe-Compose) with harnessing-level recursion that distills reusable harness components and lessons into an evolving improvement library across tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn