Controlled Multi-Agent Causal Benchmark for Autonomous Research Harnesses
Evaluate cross-model adversarial research agents against compute-matched single-agent and homogeneous baselines using blinded expert peer reviewers across benchmark paper tasks.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 5.1 and Section 7 note that the operational evidence reported for ARIS—such as an overnight run improving paper review scores from 5.0 to 7.5/10—is observational and derived from a single paper trajectory. The authors emphasize that isolating the true causal effect of cross-model reviewer heterogeneity from model capacity, prompt engineering, or task difficulty requires a controlled benchmark protocol.
“All reported outcomes are observational; they cannot be causally attributed to ARIS alone.”
Limitation
System efficacy claims rely entirely on observational single-run trajectory data without compute-matched baselines or blinded human expert evaluation to prove causal advantage over single-agent or same-model self-refinement.
Proposed Solution
Implement the complete benchmark protocol outlined in Appendix E using a curated corpus of 15+ paper drafts across diverse ML subfields. Benchmark five compute-matched configurations: single-model self-critique, same-model two-agent review, cross-model heterogeneous pairing (e.g. Claude executor + GPT reviewer), reversed cross-model pairing, and alternative secondary model backends. Measure issue recall, false-positive rate, actionability score, and downstream manuscript quality judged double-blinded by three independent domain experts using Krippendorff's alpha for inter-rater agreement.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Would establish the first rigorous empirical proof of whether cross-model adversarial collaboration provides statistically significant gains in autonomous research quality over simpler single-model self-refinement.
How to Get Started
How to test it: Test the hypothesis that cross-model heterogeneous reviewer pairing yields significantly higher human peer-review scores and lower claim fabrication rates than compute-matched same-model self-refinement. The baseline is same-model two-agent review. Metrics include blinded human reviewer scores (1-10 scale) and Krippendorff's alpha > 0.70. Stop if cross-model pairing demonstrates no statistically significant advantage (p > 0.05).
- Curate 15 open-source preprint drafts spanning diverse computer vision, NLP, and RL tasks along with their reference codebases.
- Build an automated evaluation driver that executes five compute-matched harness configurations under fixed random seed and token consumption constraints.
- Generate manuscript revisions and claim auditing reports across all harness conditions.
- Recruit three expert human peer reviewers per paper topic to grade the anonymized outputs using standardized ICLR evaluation rubrics.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like ResearchClawBench and AgencyBench explore benchmarking autonomous agents, specifically in scientific research contexts or long-context scenarios. However, the proposal to specifically evaluate causal multi-agent research harnesses through a controlled cross-model pairing study with double-blind expert assessment remains distinct from these general benchmarking frameworks.
- AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts · 2026 · 40 citations
- ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research · 2026 · 30 citations
We searched OpenAlex and arXiv for “autonomous research agent evaluation benchmark cross model critique” and “Controlled Multi-Agent Causal Benchmark for Autonomous Research Harnesses” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
ARIS: Autonomous Research via Adversarial Multi-Agent CollaborationRuofeng Yang, Yongcan Li, Shuai Li · 2026
This paper presents ARIS, an open-source research harness for autonomous machine learning research designed to mitigate the risk of plausible unsupported claims produced by long-horizon agents. ARIS enforces cross-model adversarial collaboration by pairing an executor LLM from one model family with a reviewer LLM from another family across five automated research workflows. The system incorporates an assurance audit cascade, persistent research wiki state, and modular Markdown skills to ensure experimental integrity and manuscript quality.
Related Ideas
- Automated Reproducibility Auditing for AI-Generated Scientific BaselinesEvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery🟡Fairly easy
- Interactive Human-in-the-Loop Refinement with Grounded Revision ConstraintsPaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing🟡Fairly easy
- Calibrating Automated Scientific Literature Review Evaluators Against Structural LLM BiasPaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing🟡Fairly easy
- Artifact-to-Claim Cross-Verification and Process Consistency Rewards for Agentic DeliverablesSage S2.1 Flash Technical Report🟡Fairly easy
- Retrieval-Augmented Soundness Verification Against External LiteraturePaperCritic: Training AI Reviewers to Assess the Soundness of Papers🔴Hard