Tuning prompts against small evaluation sets risks model overfitting

1. Perform semantic deduplication on your evaluation dataset before splitting it into training and holdout sets.
2. Keep a strict holdout evaluation set that the prompt author is not allowed to inspect.
3. Build adversarial evaluation slices using fresh failure cases to test prompt robustness.
4. Use confidence intervals and pass-at-k metrics to get a more honest assessment of tasks with multiple acceptable outputs.