Retrieval-Augmented Feedback for Code Generation
Integrate a retrieval-augmented generation system into compiler-based feedback loops to supply relevant hardware manuals during code generation mutation.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 5.1, GEPA is applied to code generation (NPUEval) using compiler errors as feedback. The authors note that 'kernel development expertise—often codified in technical manuals—can be selectively surfaced' but their current implementation is preliminary.
“Specifically, kernel development expertise—often codified in technical manuals and documentation—can be selectively surfaced by retrieving relevant manual sections based on rollout failures (e.g., compiler error messages).”
Limitation
The current feedback function relies primarily on the LLM's internal knowledge and raw error messages. When optimizing for obscure hardware (like NPUs) or complex APIs, the LLM may lack the specific syntax knowledge required to fix the error based on the trace alone.
Proposed Solution
Integrate a Retrieval-Augmented Generation (RAG) system into the feedback function (mu_f). When a compiler error occurs, use the error message to retrieve relevant snippets from the specific hardware documentation or API manual. Feed this retrieved context into the reflective mutation step to guide the LLM in generating syntactically correct kernel code.
Why This Matters
This extends GEPA from general reasoning tasks to specialized engineering domains where LLMs traditionally hallucinate, enabling automated optimization of high-performance code for new hardware architectures.
How to Get Started
How to test it: Note that the paper already implements error-driven RAG inside the feedback function mu_f for NPUEval and KernelBench in Section 5.1. To isolate and validate the RAG-in-mu_f contribution on an unseen hardware target or API, compare GEPA with RAG-augmented mu_f against GEPA with raw compiler-error mu_f (no RAG) on NPUEval and a held-out kernel suite using mean vector utilization and fast_1; stop if RAG-augmented mu_f does not improve vector utilization or fast_1 over raw compiler feedback.
- Construct a vector database containing technical manuals and API documentation for target hardware kernels.
- Modify the reflective mutation function to capture compiler error messages as search queries.
- Retrieve top-k relevant documentation snippets and append them to the prompt context sent to the large language model.
- Evaluate the updated code generation pipeline on NPUEval using syntactic correctness and execution success metrics.
Code linked to this paper on Hugging Face: github.com/gepa-ai/gepa
Suggested by AI. Adapt it to your setup.
Has This Been Done?
AgenticCANN: Automated Ascend C Operator Generation via Knowledge-Augmented Agentic Evolution explores operator generation, and Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback uses RAG with feedback for code generation. However, they do not specifically target using compiler errors to retrieve hardware documentation for iterative kernel mutation.
- AgenticCANN: Automated Ascend C Operator Generation via Knowledge-Augmented Agentic Evolution · 2026 · 4 citations
- Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback · 2026 · 5 citations
We searched OpenAlex for “Integrate a retrieval-augmented generation system into compiler-based feedback loops to supply relevant hardware manuals during code generation mutation.” and “Retrieval-Augmented Feedback for Code Generation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
GEPA: Reflective Prompt Evolution Can Outperform Reinforcement LearningLakshya A Agrawal, Shangyin Tan, Dilara Soylu +14 more · 2025
This paper introduces GEPA (Genetic-Pareto), a prompt optimization algorithm for compound AI systems that outperforms Reinforcement Learning (RL) methods like GRPO. GEPA uses natural language reflection to mutate prompts based on execution traces and maintains a Pareto frontier of diverse candidates to avoid local optima, achieving higher performance with significantly fewer rollouts.
Related Ideas
- Cross-Interface Harness Adaptation via Abstract Contract SynthesisSelf-Improvement Memory Enables Continual Recursive Self-Improvementđź”´Hard
- Hierarchical Specification-to-Patch Synthesis for Large-Scale Repository ChangesSWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizationsđźź Medium
- Structure-Aware Retrieval for Coding Agent ContextAgent READMEs: An Empirical Study of Context Files for Agentic Coding🟡Fairly easy
- Instruction-Controllable VLMs for Strict Formatting via Preference OptimizationStepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports🟢Easy
- Grammar-Constrained Command GenerationAutoDev: Automated AI-Driven Development🟡Fairly easy