Prompt & RAG Evaluation Harness
Stop shipping prompts on vibes: 400+ graded queries and a CI gate that blocks retrieval and answer-quality regressions.
Before
Every prompt or retrieval tweak on the RAG and NL-to-SQL systems was shipped on intuition. There was no way to know whether a change quietly regressed quality until users noticed.
After
An evaluation harness with ~400 graded queries, judge-LLM scoring, and a CI step that blocks regressions. It caught 6 regressions before production and made prompt iteration data-driven instead of guesswork.
Every prompt or retrieval tweak was being shipped on vibes — there was no way to measure regressions on the RAG and NL-to-SQL systems.
Open-source-flavoured eval harness with ~400 graded queries, judge-LLM scoring, and a CI step that blocks regressions in retrieval recall and answer quality.
- 1
Gold-standard sets
Phase 1Assembled ~400 graded queries across the RAG and NL-to-SQL systems — the fixed yardstick every future change is measured against.
- 2
Judge-LLM scoring
Phase 2Added an LLM judge to score answer quality and a retrieval-recall metric, so 'better' becomes a number instead of an opinion.
- 3
CI gate
Phase 3Wired the harness into CI so any prompt or retrieval change that drops recall or answer quality below threshold fails the build before it can merge.
- 4
Adoption
Phase 4 — PresentMade the harness the mandatory gate on every RAG/NL-to-SQL change. It caught 6 regressions that would otherwise have shipped.
Fixed gold-standard sets over ad-hoc spot checks
Why · A stable, graded set is the only way to compare two versions honestly. Ad-hoc testing hides regressions rather than surfacing them.
Judge-LLM plus a hard retrieval metric
Why · Answer-quality judging is fuzzy; retrieval recall is not. Pairing a soft judge with a hard metric catches both reasoning and retrieval regressions.
Block in CI, not in review
Why · Regressions caught by a human reviewer are caught inconsistently. A CI gate makes quality non-negotiable and removes it from opinion.
- Caught 6 regressions before they hit production
- Made prompt iteration data-driven instead of intuition-driven
- Adopted as the gate on every RAG/NL-to-SQL change
- 01 · Evaluation infrastructure is the highest-leverage thing you can build for a RAG system — build it early.
- 02 · You can't improve what you don't measure, and 'the answers feel better' isn't a measurement.
- 03 · A judge-LLM plus one hard metric beats either alone.
- 04 · The moment quality lives in CI, prompt iteration stops being scary.
Ask anything about Prompt & RAG Evaluation Harness
AI scoped to this project · GPT-OSS 120B