Sensitivity-Aware Composite Indexing and Item-Response Calibration Across Heterogeneous Benchmarks
Calibrate composite benchmark indices using item-response modeling and leave-one-out weighting sensitivity so model rankings reflect true clinical capability rather than arbitrary portfolio weights.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 4.3 and Table 1, the InclusionMed Index aggregates five benchmarks via an unweighted arithmetic mean (20% weight each), even though task counts range from 22 (ALE-CLI) to 525 (HealthBench Professional), metrics mix length-adjusted partial-credit rubrics with strict all-checkpoint pass@1 and perfect-grade rates, and two benchmarks share the same cell.
“Future sensitivity analyses could vary benchmark weights and omit one benchmark at a time to examine which model comparisons remain stable and which depend strongly on the portfolio [41].”
Limitation
Equal-weight averaging over uncalibrated, scale-heterogeneous benchmarks allows high-variance or small-sample suites (such as 22-task ALE-CLI or strict binary pass@1 suites) to disproportionately swing overall rankings and obscures statistical uncertainty.
Proposed Solution
Apply a multi-dimensional Item Response Theory (IRT) or Rasch-style latent capability model across task-level outcomes (1,034 tasks) to jointly estimate task difficulty, discrimination, and model latent ability across roles and capabilities, supplemented by Dirichlet-sampled weight sensitivity analysis, leave-one-benchmark-out stability checks, and bootstrap confidence intervals over tasks and runs.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Principled calibration and uncertainty reporting would prevent ranking artifacts driven by benchmark size or strictness and show stakeholders which model comparisons are statistically robust.
How to Get Started
How to test it: Test whether IRT-calibrated capability scores and leave-one-out weight perturbations identify unstable rank pairs among the 22 models compared to the equal-weight Index using Kendall's tau and bootstrap rank-overlap intervals. Stop if task-level score matrices cannot be obtained or if latent model fit fails posterior predictive checks.
- Retrieve or reconstruct per-task binary and continuous score tables for all 22 models across HealthBench Professional, PhysicianBench, CHI-Bench, and ALE-CLI v1.0.
- Implement leave-one-benchmark-out and 10,000-draw Dirichlet weight perturbation analyses to compute rank-reversal probabilities for all model pairs, especially top-5 ties such as Opus 5.5 and Fable 5.1.
- Fit a two-parameter logistic/graded-response IRT model across the 1,034 tasks to separate task difficulty from model role/capability profiles and compare the resulting Pareto frontier against Figure 5.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like Auditing LLM Benchmarks with Item Response Theory and Validity-Aware Assessment of LLM Value Orientation demonstrate the use of IRT to calibrate benchmarks and evaluate model capabilities. However, these works do not specifically detail the proposed multi-dimensional IRT approach applied to a large-scale set of 1,034 heterogeneous tasks combined with bootstrap stability checks.
- Auditing LLM Benchmarks with Item Response Theory · 2026 · 5 citations
- Validity-Aware Assessment of LLM Value Orientation via Subset Item Response Theory · 2026 · 0 citations
- Heterogeneous Judge-Aware Ranking with Sensitivity, Disagreement, and Confidence · 2026 · 2 citations
We searched OpenAlex and arXiv for “item response theory LLM benchmark aggregation sensitivity analysis” and “Sensitivity-Aware Composite Indexing and Item-Response Calibration Across Heterogeneous Benchmarks” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
InclusionMed: Benchmarking AI for Inclusive HealthcareThe InclusionMed Collaboration · 2026
InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.
Related Ideas
- Controlled Multi-Agent Causal Benchmark for Autonomous Research HarnessesARIS: Autonomous Research via Adversarial Multi-Agent Collaboration🟠Medium
- Difficulty- and Saturation-Aware Adaptive Task-Level Recursion BudgetingHarness^2: Recursive Agent Harnessing for an Open World🟡Fairly easy
- Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable SelectionHarness^2: Recursive Agent Harnessing for an Open World🟢Easy
- Conformal Multimodal Outcome Verification to Mitigate Premature Completion BiasEnvisioning Agentic Robotics: A New Era of Robotic Systems🟡Fairly easy
- Artifact-to-Claim Cross-Verification and Process Consistency Rewards for Agentic DeliverablesSage S2.1 Flash Technical Report🟡Fairly easy