Code-Augmented Counter-Example Verification for Auto-Graders
Integrate an automated code-execution loop using Python and SymPy to verify mathematical derivation steps and catch logical errors in auto-graders.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Appendix B.6 (Table 12), the ProofAutoGrader falsely marked a solution as correct because it failed to detect a 'specious' logical error (assuming that if A≥B and A≥C, then B≥C). It relied on text-based plausibility rather than mathematical validity.
“To address these limitations, we use large language models as automated verifiers for model answers on IMO-AnswerBench.”
Limitation
The current text-based ProofAutoGrader struggles to identify subtle logical fallacies or algebraic inconsistencies that look plausible in natural language but are mathematically invalid.
Proposed Solution
enhance the AutoGrader by integrating a code-execution loop (Python/SymPy) specifically for verification. When the model evaluates a derivation step (e.g., an inequality transition), it should generate a script to search for numerical counter-examples within the domain. If a counter-example is found, the step is automatically flagged as incorrect, overriding the text-based judgment.
Why This Matters
Improving auto-grader reliability is critical for scaling oversight. A 96% correlation is insufficient for high-stakes math where a single logical flaw invalidates the entire proof.
How to Get Started
How to test it: We hypothesize that integrating a SymPy verification loop will reduce logical errors in derivations compared to text-only baselines. We will measure the exact match accuracy on IMO-AnswerBench. If the execution loop introduces syntax timeouts or fails to improve accuracy over the base model, we will stop.
- Implement a secure sandboxed execution environment capable of running Python and SymPy scripts.
- Create a verification module that extracts mathematical inequality transitions and formulates search scripts for numerical counter-examples.
- Integrate the execution loop into the grading pipeline to override text-based plausibility scores when a counter-example is found.
- Test the updated auto-grader on benchmark datasets containing known specious logical errors.
Code linked to this paper on Hugging Face: github.com/google-deepmind/superhuman
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers cover code-augmented counter-example verification for auto-graders or mathematical inequality steps. Papers like 'Can LLMs Reliably Grade Olympiad Proofs?' and 'Harnessing Code Agents for Automatic Software Verification' discuss general grading or software verification, but do not specifically implement a code-execution loop to verify inequality transitions or derivation steps.
- Harnessing Code Agents for Automatic Software Verification · 2026 · 3 citations
- Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs · 2026 · 0 citations
We searched OpenAlex for “Integrate an automated code-execution loop using Python and SymPy to verify mathematical derivation steps and catch logical errors in auto-graders.” and “Code-Augmented Counter-Example Verification for Auto-Graders” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
Towards Robust Mathematical ReasoningThang Luong, Dawsen Hwang, Hoang H. Nguyen +17 more · 2025
This paper introduces IMO-Bench, a challenging benchmark suite designed to evaluate mathematical reasoning at the International Mathematical Olympiad (IMO) level, comprising tasks for short-answer verification (IMO-AnswerBench), proof writing (IMO-ProofBench), and proof grading (IMO-GradingBench). The authors demonstrate that current frontier models, while improving, still struggle with robustification (reformulated problems) and subtle logical fallacies in proofs, proposing a new gold-standard model (Gemini Deep Think) that achieves significant gains through agentic search and verification.
Related Ideas
- Formal Claim-to-Proof Consistency Verification for Detecting Mathematical Overclaims in Scientific ProseK2-Horizon: A Frontier Fleet of Open Source Language Models🟠Medium
- Numerical Stability-Aware RL for Dry ExperimentsProbing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows🔴Hard
- Executable Programmatic Verifiers for Robust Micro-Skill Lifecycle ManagementTokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents🟡Fairly easy
- Mitigating RL-Induced Style Narrowing and Hallucinated Notation CritiquesPaperCritic: Training AI Reviewers to Assess the Soundness of Papers🟡Fairly easy
- Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness EditsHarness^2: Recursive Agent Harnessing for an Open World🟠Medium