Skip to content
AI-generated research idea🟡Fairly easy🔧Minor TweakRelated work found

Full-Trajectory Deployment Cost and Retry-Overhead Profiling for Medical AI Agents

Profile end-to-end agent deployment costs including failed retries, external judges, and sandbox overhead so healthcare teams can compare true operational affordability.

Time

several weeks

Compute level

Single GPU or less

Resources

API credits to re-run a representative subset of the 22 models across PhysicianBench, CHI-Bench, and ALE-CLI v1.0, plus standard cloud CPU/container hosting for FHIR and CLI sandboxes

What the paper used

Evaluated 22 commercial and open-weight LLM releases via API and agent harnesses across 1,034 tasks from five benchmarks (up to 8 runs per task on 525 cases, 3 runs on 75 cases, and 1 run on 434 cases), plus GPT-5.4 and GPT-6.1 Sol as automated judges, with cited per-task API costs from $0.064 to $4.12; local GPU hardware is not stated in the paper.

Prerequisites

LLM agent serving, API cost telemetry, Docker/FHIR sandboxes, Pareto optimization

Scope

modify existing code

The Gap

In Sections 4.4, 5.4, and 6.3 (and Appendix A.2), inference costs used to construct the Pareto frontier are computed solely from recorded input, output, and cached token prices on valid runs, omitting API/safety-block retries (up to 10 retries on HealthBench Professional, unlimited retries until valid on PhysicianBench and CHI-Bench), task timeouts (up to 7,200 s on ALE-CLI), and tool/sandbox/judge charges.

“They exclude failed runs and retries for the four public benchmarks, as well as judge, search, tool, sandbox, and internal platform charges (Appendix A.2), and therefore do not represent total deployment expenditure.”
From the paper, Section 6.3, page 15. Matched against the PDF text.

Limitation

Excluding failed attempts, retries, tool execution overhead, and automated judge fees underestimates real-world deployment costs and may misclassify brittle models with high retry or timeout rates as Pareto-optimal.

Proposed Solution

Instrument the FHIR and CLI evaluation harnesses with OpenTelemetry-style full-trajectory billing that logs token consumption across every retry and safety-block fallback, wall-clock container/sandbox compute-seconds, external search/FHIR API calls, and LLM-judge token costs. Compute risk-adjusted cost-per-successful-task metrics and re-derive the performance-cost Pareto frontier under realistic hospital concurrency and SLA latency constraints.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Full-trajectory accounting should give clinical IT teams accurate budget projections and expose hidden reliability taxes in agentic healthcare deployments.

How to Get Started

How to test it: Test the hypothesis that including retry, timeout, and sandbox overhead shifts at least two models off the current five-model Pareto frontier by measuring full cost per successful task across the top 10 models on PhysicianBench and CHI-Bench. Abandon full-suite re-runs if initial pilot traces show retry/sandbox overhead adds less than 5% variance to token-only cost.

  1. Fork the CHI-Bench, PhysicianBench, and ALE-CLI harness wrappers to wrap every API call, retry, timeout, and Docker container execution with token, latency, and compute-second loggers.
  2. Re-run the five Pareto-frontier models plus five mid-tier models on CHI-Bench (75 tasks) and PhysicianBench (100 tasks) to capture failed-run and retry telemetry.
  3. Compare token-only mean cost per task against total cost per successful task (including retries and container runtime) and test whether the Pareto frontier in Figure 5 changes.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Cost-Utility Alignment in LLM Agent Trajectories directly addresses profiling and evaluation of agent costs and trajectories in clinical contexts. The current proposal extends this by integrating FHIR-specific infrastructure and hospital-level SLA latency constraints, which remain an open research frontier in medical agent deployment.

We searched OpenAlex and arXiv for “LLM agent deployment cost retries sandbox latency Pareto” and “Full-Trajectory Deployment Cost and Retry-Overhead Profiling for Medical AI Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

InclusionMed: Benchmarking AI for Inclusive Healthcare

The InclusionMed Collaboration · 2026

InclusionMed introduces an open medical AI evaluation framework featuring the nine-cell InclusionMed Atlas (three healthcare roles by three AI capabilities) and the InclusionMed Index, which benchmarks 22 frontier language models across five audited evaluations. The study reveals substantial rank reversals across roles and capabilities, weaker clinical decision-making and action execution relative to patient communication, and up to 24-fold inference cost disparities among similarly scoring models.

Related Ideas

Is this a good idea?

Share onXLinkedIn