Skip to content
AI-generated research idea🟡Fairly easy🔧Minor TweakRelated work found

Automated Reproducibility Auditing for AI-Generated Scientific Baselines

Deploy an independent reviewer agent to cross-reference generated experimental setups against original baseline papers before manuscript finalization.

Time

4-6 weeks (1-2 weeks to build literature-grounded config auditing, 1-2 weeks to execute the 6 end-to-end pipelines with re-runs, and 1-2 weeks for blind human and AI review).

Compute level

Single GPU or less

Resources

API credits for Gemini-2.5-Pro and Claude-4.5-Haiku across 6 end-to-end tasks plus baseline re-executions (4 parallel workers), Semantic Scholar and full-text PDF retrieval/parsing infrastructure, Ollama (mxbai-embed-large), sandboxed GPU/CPU compute for re-running flagged baselines, and 3 PhD reviewers for blind scoring.

What the paper used

Uses Semantic Scholar API, Gemini-2.5-Pro (ideation and manuscript authoring), Claude-4.5-Haiku (code generation), Gemini-3-flash (automatic evaluation), and mxbai-embed-large via Ollama (memory retrieval). Scale: 30 research queries (N_I=21 candidate ideas with 3 parallel workers; 4 parallel experiment workers with 62 max attempts per query across 4 stages: 20, 12, 12, 18), 6 end-to-end papers, and 3 PhD annotators on 120 pairs. Specific GPU/CPU hardware specs are not stated in the paper.

Prerequisites

basic ML knowledge

Scope

modify existing code

The Gap

In the end-to-end evaluation case studies (Appendix E), reviewer feedback indicated issues with protocol-faithful baseline design, internal consistency, and reproducibility-complete reporting in the generated manuscripts.

“At the same time, they underscore that sustained reviewer confidence depends on internal consistency, protocol-faithful baseline design, and reproducibility-complete reporting, pointing to natural directions for further system-level refinement.”
From the paper, Appendix E.2, page 15. Matched against the PDF text.

Limitation

The system currently lacks a rigorous, automated auditing mechanism to ensure that the experimental setups for baselines perfectly match standard literature protocols, occasionally leading to unfair comparisons or omitted experimental details.

Proposed Solution

Introduce an independent 'Reviewer/Auditor Agent' that runs prior to paper finalization. This agent would be explicitly prompted to cross-reference the generated experimental setup against retrieved original baseline papers, flagging unfair configuration discrepancies and forcing the Engineer Agent to re-run baselines under identical constraints.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Ensuring rigorous, fair comparisons and complete reproducibility is crucial for maintaining the credibility and academic integrity of AI-generated scientific outputs.

How to Get Started

How to test it: Hypothesis: Running an automated Reviewer/Auditor Agent that cross-checks baseline code configs against retrieved literature prior to manuscript drafting will reduce baseline-fairness and reproducibility critiques. Baseline: Original EvoScientist end-to-end pipeline on the 6 ICAIS 2025 topics, evaluated by blind PhD and AI reviewers on protocol-fairness error count and reproducibility completeness; stop if reviewer-flagged protocol discrepancies do not drop by at least 50% or false-positive audits exhaust the Engineer Agent's attempt budget on over 20% of runs.

  1. Create a repository of baseline reference papers and their standard configuration parameters.
  2. Develop a protocol-checking agent prompted to identify discrepancies between generated code and original baseline specifications.
  3. Implement an automated re-run constraint that forces the engineering agent to fix unfair configuration mismatches.
  4. Evaluate the auditing framework on a test set of generated scientific manuscripts.

Code linked to this paper on Hugging Face: github.com/EvoScientist/EvoScientist

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works such as RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers? and audit-closed-ai-scientist deal with agentic auditing and reproducibility of claims. However, they do not specifically implement a pre-finalization Auditor Agent that actively forces an Engineer Agent to re-run baselines under identical constraints based on retrieved setup discrepancies.

We searched OpenAlex for “Deploy an independent reviewer agent to cross-reference generated experimental setups against original baseline papers before manuscript finalization.” and “Automated Reproducibility Auditing for AI-Generated Scientific Baselines” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery

Yougang Lyu, Xi Zhang, Xinhao Yi +9 more · 2026

This paper introduces EvoScientist, a multi-agent framework that automates the end-to-end scientific discovery process by leveraging self-evolving AI agents. Its primary contribution is the integration of persistent memory modules that allow the system to continuously learn from past idea validations and code execution outcomes, significantly improving the novelty, feasibility, and success rates of its generated research.

Related Ideas

Is this a good idea?

Share onXLinkedIn