Mitigating Few-Shot Overfitting via Dynamic Sample Rotation
Prevent few-shot prompt optimization overfitting by dynamically rotating the batch of unlabelled question instances at every iteration.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In 'Ablation Study on Iteration Rounds' (Appendix A.4.3), the authors observe that performance peaks around iteration 7-10 and then degrades. They attribute this to 'overfitting to the sample questions' (the 3 specific unlabelled inputs used in the loop).
Limitation
Because the optimization loop iterates heavily on a fixed, tiny set of 3 samples, the generated prompt eventually becomes a 'lookup table' or heuristic specifically tuned to solve those 3 inputs, losing generalization power on the unseen test set.
Proposed Solution
Implement Dynamic Sample Rotation. Instead of fixing the dataset Q at initialization (Algorithm 1, Line 1), sample a *new* batch of questions Qt from the unlabelled dataset D at every iteration t. This forces the optimizer to find a prompt that generalizes across the distribution rather than overfitting to specific instances.
Why This Matters
This stabilizes the training process, preventing the 'collapse' observed after 10 iterations and potentially allowing for longer optimization runs that yield even higher quality prompts.
How to Get Started
- Review the existing prompt optimization loop script and locate the static dataset initialization.
- Replace the fixed subset with a dynamic data loader that draws fresh question batches from the unlabelled pool at each iteration.
- Run tuning experiments and track validation performance across iteration rounds to identify potential regression.
- Compare generalization capability against a static-dataset baseline on held-out test sets.
Code linked to this paper on Hugging Face: github.com/geekan/metagpt
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The provided literature covers few-shot incremental learning and few-shot segmentation, such as Few-Shot Incremental Learning with Continually Evolved Classifiers and Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining. None of these listed works propose dynamic sample rotation from an unlabeled dataset across iterations specifically to prevent overfitting in prompt optimization algorithms.
- Few-Shot Incremental Learning with Continually Evolved Classifiers · 2021 · 535 citations
- Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining · 2024 · 76 citations
- Dynamic Distribution Calibration Method for Few-Shot Learning · 2024 · 0 citations
We searched OpenAlex for “Prevent few-shot prompt optimization overfitting by dynamically rotating the batch of unlabelled question instances at every iteration.” and “Mitigating Few-Shot Overfitting via Dynamic Sample Rotation” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
Self-Supervised Prompt OptimizationJinyu Xiang, Jiayi Zhang, Zhaoyang Yu +6 more · 2025
The paper introduces Self-Supervised Prompt Optimization (SPO), a framework designed to automate the improvement of prompts for Large Language Models (LLMs) without relying on ground truth data or human feedback. SPO leverages the inherent capability of LLMs to distinguish better answers through pairwise comparisons (Output vs. Output) to guide the optimization process, achieving state-of-the-art performance at a fraction of the cost of existing methods.
Related Ideas
- Stabilizing On-Policy Distillation via Critic RegularizationLiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation🟡Fairly easy
- Differentiable Relaxation of Written Associative Memories for Few-Shot Fine-TuningWeight-Space Programming Doesn't Need Transformers🟡Fairly easy
- Accelerating 3D-Guided RL Fine-Tuning with Trajectory DistillationGeometry-Guided Reinforcement Learning for Multi-view Consistent 3D Scene Editing🔴Hard
- Dynamic Self-Play Fine-Tuning PipelineLlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models🟠Medium
- Invariance Training via Adversarial RobustificationTowards Robust Mathematical Reasoning🟠Medium