Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Code-Augmented Counter-Example Verification for Auto-Graders

Integrate an automated code-execution loop using Python and SymPy to verify mathematical derivation steps and catch logical errors in auto-graders.

Time

2 weeks

Compute level

Multiple GPUs

Resources

A multi-GPU node or API access to a strong frontier model with Python execution sandbox capabilities.

What the paper used

Gemini 2.5 Pro with 32K token thinking budget per call

Prerequisites

basic ML knowledge, prompt engineering, Python/SymPy

Scope

modify existing code

The Gap

In Appendix B.6 (Table 12), the ProofAutoGrader falsely marked a solution as correct because it failed to detect a 'specious' logical error (assuming that if A≥B and A≥C, then B≥C). It relied on text-based plausibility rather than mathematical validity.

“To address these limitations, we use large language models as automated verifiers for model answers on IMO-AnswerBench.”
From the paper, Section 2.3, page 4. Matched against the PDF text.

Limitation

The current text-based ProofAutoGrader struggles to identify subtle logical fallacies or algebraic inconsistencies that look plausible in natural language but are mathematically invalid.

Proposed Solution

enhance the AutoGrader by integrating a code-execution loop (Python/SymPy) specifically for verification. When the model evaluates a derivation step (e.g., an inequality transition), it should generate a script to search for numerical counter-examples within the domain. If a counter-example is found, the step is automatically flagged as incorrect, overriding the text-based judgment.

Why This Matters

Improving auto-grader reliability is critical for scaling oversight. A 96% correlation is insufficient for high-stakes math where a single logical flaw invalidates the entire proof.

How to Get Started

How to test it: We hypothesize that integrating a SymPy verification loop will reduce logical errors in derivations compared to text-only baselines. We will measure the exact match accuracy on IMO-AnswerBench. If the execution loop introduces syntax timeouts or fails to improve accuracy over the base model, we will stop.

  1. Implement a secure sandboxed execution environment capable of running Python and SymPy scripts.
  2. Create a verification module that extracts mathematical inequality transitions and formulates search scripts for numerical counter-examples.
  3. Integrate the execution loop into the grading pipeline to override text-based plausibility scores when a counter-example is found.
  4. Test the updated auto-grader on benchmark datasets containing known specious logical errors.

Code linked to this paper on Hugging Face: github.com/google-deepmind/superhuman

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 6 results

None of the listed papers cover code-augmented counter-example verification for auto-graders or mathematical inequality steps. Papers like 'Can LLMs Reliably Grade Olympiad Proofs?' and 'Harnessing Code Agents for Automatic Software Verification' discuss general grading or software verification, but do not specifically implement a code-execution loop to verify inequality transitions or derivation steps.

We searched OpenAlex for “Integrate an automated code-execution loop using Python and SymPy to verify mathematical derivation steps and catch logical errors in auto-graders.” and “Code-Augmented Counter-Example Verification for Auto-Graders” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

Towards Robust Mathematical Reasoning

Thang Luong, Dawsen Hwang, Hoang H. Nguyen +17 more · 2025

This paper introduces IMO-Bench, a challenging benchmark suite designed to evaluate mathematical reasoning at the International Mathematical Olympiad (IMO) level, comprising tasks for short-answer verification (IMO-AnswerBench), proof writing (IMO-ProofBench), and proof grading (IMO-GradingBench). The authors demonstrate that current frontier models, while improving, still struggle with robustification (reformulated problems) and subtle logical fallacies in proofs, proposing a new gold-standard model (Gemini Deep Think) that achieves significant gains through agentic search and verification.

Related Ideas

Is this a good idea?

Share onXLinkedIn