12 August 2026
A retrieval system you cannot score is a retrieval system you cannot maintain
Why the evaluation harness has to be built alongside the retriever, not after it, and what to put in it.
The problem with shipping a RAG system that works
A retrieval system almost always works when you first demo it. You ask the questions you had in mind while building it, and the answers come back with sensible citations. Then someone changes the chunk size, or the embedding model gets a new version, or a hundred new documents land in the corpus, and nobody can tell you whether the system got better or worse.
Not "nobody has checked yet". Nobody can tell you, because there is nothing to check against.
What the harness actually needs
Three things, and they are less work than they sound.
A held-out question set that reflects real questions. Not questions written by the person who built the retriever, because those questions are shaped by what the retriever can already do. Take them from the people who will use the system, before you show them anything.
A metric that matches the decision. Retrieval quality and answer quality are different failures with different fixes. If the right passage was never retrieved, no amount of prompt work will save the answer. Measure whether the correct passage appears in the retrieved set at all, separately from whether the answer built on it was any good.
A number you can re-run in one command. A harness that takes an afternoon to run is a harness that gets run once. The point is that it runs on every change, so a regression is caught by the person who caused it, on the day they caused it.
The part people skip
Refusals. A retrieval system that answers everything is not grounded, it is guessing politely. Your evaluation set needs questions the corpus genuinely cannot answer, and the correct behaviour on those is a refusal. Score it. A system that never refuses will happily invent a citation.
What this buys you
The ability to say, in a sentence, whether the change you are about to ship helps. That is the whole return, and it is the difference between a system you maintain and a system you are afraid to touch.