Cross-Family Harness Standardization and Multi-Judge Debiasing in Agentic Clinical Evaluation
Standardize agent execution harnesses across model families and evaluate with multi-judge ensembles so workflow scores measure model capability rather than tooling artifacts.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Appendix A.2 and Table S2, agency benchmarks (CHI-Bench and ALE-CLI v1.0) evaluate GPT models with Codex, Gemini models with Gemini CLI, and all other 16 models with Claude Code (furthermore using Claude Code 2.1.280 for Opus 5.5 versus 2.1.170 for others). Separately, Appendix C.3 evaluates InclusionMed-DxFiles solely using GPT-6.1 Sol as the judge, even though GPT-6.1 Sol is a ranked top-5 competitor in the Index.
“For both CHI-Bench and ALE-CLI v1.0: Health and Medicine, we used Codex for the four GPT releases, Gemini CLI for the two Gemini releases, and Claude Code for the remaining 16 releases.”
Limitation
Confounding model families with vendor-specific CLI harnesses makes it unclear whether agency gaps (e.g., GPT-6 Luna ranking 8th in reasoning but 20th in agency) stem from the base model or the scaffold, while single-model judging with an evaluated model risks self-preference bias.
Proposed Solution
Run a controlled factorial evaluation using a single open-source, model-agnostic CLI and tool-use harness (e.g., an OpenHands or standardized ReAct/MCP scaffold) across all flagship families alongside vendor-native harnesses to decompose model versus harness variance. Replace single-model judging on diagnostic reasoning and rubric tasks with a cross-family judge panel (e.g., combining Claude, Gemini, and GPT judges with blinded outputs and human-adjudicated calibration checks).
Why This Matters
Isolating base-model agency from harness design and removing single-judge bias would make leaderboard comparisons fair across providers and prevent over-crediting proprietary scaffolding.
How to Get Started
How to test it: Measure the variance in CHI-Bench pass@1 explained by model versus harness (standardized open harness vs Codex/Gemini CLI/Claude Code) across six flagship models via two-way ANOVA, and measure judge self-preference bias on diagnostic cases using human physician ground truth. Conclude the confound is negligible if rank correlation across harnesses exceeds 0.95 and judge-family bias is under 2 percentage points.
- Implement a unified open-source CLI/MCP agent adapter that exposes identical prompts, bash/file tools, and retry logic to GPT-6 Astra, Claude Opus 5.5, Gemini 4 Argon, DeepSeek V4.1 Flash, and Qwen3.8 Max.
- Evaluate all selected models on CHI-Bench (75 tasks) and ALE-CLI v1.0: Health and Medicine (22 tasks) under the unified harness and compare against the Table S2 vendor-harness scores.
- Run a cross-family three-judge ensemble (GPT, Claude, Gemini) on a 100-case stratified sample of HealthBench Professional and open NEJM MGH diagnostic cases to quantify inter-judge agreement and family self-preference.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Research such as Better Deck or Different Judge? and Agreement by Construction explore the role of judges and harnesses in evaluation, specifically within niche or corporate contexts. While these acknowledge the impact of judging variance, they do not provide a standard cross-family framework specifically tailored to the clinical reasoning and multi-agent scaffold validation described here.
- Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking · 2026 · 0 citations
- Agreement by Construction: A Deterministic Filter Inflates Inter-Judge Reliability in LLM-Judge Evaluation · 2026 · 0 citations
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods · 2024 · 893 citations
We searched OpenAlex and arXiv for “LLM agent harness confounding self-preference judge bias clinical benchmark” and “Cross-Family Harness Standardization and Multi-Judge Debiasing in Agentic Clinical Evaluation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
InclusionMed: Benchmarking AI for Inclusive HealthcareThe InclusionMed Collaboration · 2026
InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.
Related Ideas
- Heterogeneous Multi-Reviewer Ensemble Routing for Autonomous Paper RefinementARIS: Autonomous Research via Adversarial Multi-Agent Collaboration🟡Fairly easy
- End-to-End Benchmarking of Native Scientific Perception Within Interactive Agentic WorkflowsSage S2.1 Flash Technical Report🟠Medium
- Programmatic and Decomposed Soundness Verification Without Frontier Distillation BottlenecksPaperCritic: Training AI Reviewers to Assess the Soundness of Papers🟠Medium
- Domain-Specific Evaluator Calibration via Constitutional PrinciplesSelf-Supervised Prompt Optimization🟡Fairly easy
- Controlled Multi-Agent Causal Benchmark for Autonomous Research HarnessesARIS: Autonomous Research via Adversarial Multi-Agent Collaboration🟠Medium