Executable Programmatic Verifiers for Robust Micro-Skill Lifecycle Management
Replace string-based regular expression detectors with sandboxed executable Python or Bash assertion scripts to robustly verify candidate micro-skills.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 3.1 and Appendix I.1 (Example c), the LLM proposer generates regex-based detectors (scope, confirm, contradict) to evaluate candidate skills. In Example (c), a false skill advising the agent to invoke pytest on SymPy was confirmed across several tasks because the confirm regex matched the command string execution rather than checking for a zero exit status or successful test discovery.
“The skill was nonetheless activated, because its confirming detector matches the pytest command itself rather than a passing result, so the agent’s failed attempts on tasks 2–5 counted as confirmations.”
Limitation
Surface-level regular expression matching over raw bash commands and outputs is susceptible to false confirmations (e.g., matching a command invocation that resulted in a silent failure or missing module), leading to the activation of erroneous instructions that waste agent tokens before retirement.
Proposed Solution
Replace string-based regular expression detectors with sandboxed executable Python or Bash assertion scripts. When a candidate micro-skill is proposed, have the proposer synthesize an assertion script that checks the process exit status ($?), parse structured output formats (JUnit XML, TAP, or python traceback ASTs), and verify file system mutations directly within the task container. Only promote candidate skills if the programmatic verifier confirms error-free execution.
Why This Matters
Prevents toxic skill injections that degrade agent reasoning and force costly recovery cycles, increasing the precision of micro-skill injection and preventing unforced regressions in Pass@1.
How to Get Started
How to test it: Hypothesis: replacing regex detectors with proposer-synthesized executable assertion scripts (checking exit status, structured test output, and filesystem state) eliminates false micro-skill activations compared to standard TokenEconomizer. Measure retired-skill rate, cache-aware cost change, and Pass@1 across the 277 SWE-Bench tasks; stop if programmatic verification overhead offsets token savings or fails to reduce the ~20% skill retirement rate without hurting Pass@1.
- Audit existing regex-based skill detectors in the baseline codebase to catalogue failure modes and false positives.
- Build a secure Docker container sandbox environment capable of executing untrusted assertion scripts with strict resource limits.
- Implement a script synthesizer module that translates natural language proposal rules into executable Bash or Python assertions parsing JUnit XML or process exit codes.
- Compare the promotion accuracy of executable verifiers against regex baselines on SWE-Bench tasks.
Code linked to this paper on Hugging Face: github.com/microsoft/TokenEconomizer
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed search results address replacing string-based regular expression detectors with sandboxed executable Python or Bash assertion scripts for micro-skill verification. While works like ShellVis explore sandboxed live programming for shell scripts, they do not target programmatic micro-skill lifecycle management and validation.
We searched OpenAlex and arXiv for “Replace string-based regular expression detectors with sandboxed executable Python or Bash assertion scripts to robustly verify candidate micro-skills.” and “Executable Programmatic Verifiers for Robust Micro-Skill Lifecycle Management” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding AgentsAaron J. Li, Weijia Xu, Chandan Singh +8 more · 2026
This paper addresses the excessive token costs incurred by frontier coding agents when executing sequential software engineering tasks within shared environments. The authors introduce TokenEconomizer, an external, black-box controller that monitors execution traces to adaptively inject environment-specific micro-skills, prune unused tool definitions from agent contexts, and interrupt unproductive failure loops. Evaluated across Claude Code, Codex, and GitHub Copilot CLI on benchmarks including SWE-Bench, Web-Bench, and security auditing datasets, TokenEconomizer achieves an 18.4% to 30.9% token cost reduction while preserving or improving task success rates.
Related Ideas
- Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness EditsHarness^2: Recursive Agent Harnessing for an Open World🟠Medium
- Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable SelectionHarness^2: Recursive Agent Harnessing for an Open World🟢Easy
- Automated Tool-Schema Inference for Zero-Config Typed-Identity Egress GuardsSPILLWAY: Separating Target Admission from Sanitization to Defend Tool-Using Agents against Prompt Injection🟡Fairly easy
- Grammar-Constrained Command GenerationAutoDev: Automated AI-Driven Development🟡Fairly easy
- Conformal Multimodal Outcome Verification to Mitigate Premature Completion BiasEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy