Domain-Specific Evaluator Calibration via Constitutional Principles
Incorporate domain-specific constitutional guidelines retrieved via RAG into evaluation prompts to improve pairwise model assessment in specialized technical fields.
Source paper
Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong, Chenglin Wu, Yuyu Luo
Generated by AI. The paper’s authors did not write these.
Incorporate domain-specific constitutional guidelines retrieved via RAG into evaluation prompts to improve pairwise model assessment in specialized technical fields.
Optimize prompts for multiple language models simultaneously using an adversarial minimax objective that maximizes the minimum win-rate across an ensemble.
Prevent few-shot prompt optimization overfitting by dynamically rotating the batch of unlabelled question instances at every iteration.
The paper introduces Self-Supervised Prompt Optimization (SPO), a framework designed to automate the improvement of prompts for Large Language Models (LLMs) without relying on ground truth data or human feedback. SPO leverages the inherent capability of LLMs to distinguish better answers through pairwise comparisons (Output vs. Output) to guide the optimization process, achieving state-of-the-art performance at a fraction of the cost of existing methods.
SPO operates on an iterative 'Optimize-Execute-Evaluate' loop. It starts with an initial prompt and a small set of unlabelled questions. In each iteration, an LLM optimizer generates a new prompt variant. This prompt is executed to generate outputs. Crucially, instead of using ground truth, an 'LLM-as-a-judge' performs pairwise comparisons between the new outputs and the previous best outputs to determine the winner. This self-generated signal guides the next round of prompt refinement.