Home / Lab / RAG Isn't a Vector Database — It's an E…

Engineering 2 min read

RAG Isn't a Vector Database — It's an Evaluation Discipline

Everyone ships the pipeline. Almost nobody ships the harness that proves it works. Here's the eval setup we refuse to launch without.

Abstract cover artwork for this article

Ask a team how their RAG system is doing and you’ll usually get a demo, not a number. The pipeline exists — embeddings, vector store, a prompt that says “answer from the context” — but nobody can tell you the retrieval hit rate on their actual question distribution. That’s not an engineering gap. That’s flying blind.

The pipeline is the easy part

You can stand up a working retrieval pipeline in an afternoon. Chunk the documents, embed them, wire up top-k search, stuff the results into a prompt. It will look great on the three questions you try in the demo.

Then a user asks something phrased differently from how the documents phrase it. Or asks about a topic that spans two chunks. Or asks something the corpus genuinely doesn’t cover — and instead of saying so, the system confabulates a confident answer from the nearest-miss chunks it retrieved.

What we build before we build the pipeline

Every RAG engagement at Nexes starts with a golden set: 200–500 real questions with verified answers and — critically — the exact source passages that justify them. We collect them from support tickets, chat logs, and interviews with the people who currently answer these questions by hand.

That set gives us three numbers we track on every change:

  • Retrieval hit rate — is the right passage in the top-k results?
  • Answer faithfulness — does the generated answer actually follow from the retrieved context?
  • Refusal correctness — when the corpus has no answer, does the system say so?
def evaluate(pipeline, golden_set):
    for case in golden_set:
        result = pipeline.run(case.question)
        yield {
            "hit": case.source_id in result.retrieved_ids,
            "faithful": judge_faithfulness(result.answer, result.context),
            "refused_correctly": case.unanswerable == result.refused,
        }

The third metric is the one teams skip, and it’s the one that determines whether users trust the system after week two.

Chunking is a domain decision

Generic 512-token chunking destroys contracts, wrecks tables, and severs clauses from their conditions. On a recent logistics engagement, moving from fixed-size to clause-boundary chunking raised retrieval hit rate by 19 points — before we touched the embedding model. Look at your documents. Their structure is telling you where the seams are.

Ship the harness, then iterate forever

The eval harness gates every deploy, and it never stops growing: every escalation, every thumbs-down, every “that’s not quite right” becomes a new golden case. Six months in, the corpus has changed, the questions have drifted, and the system is still improving — because improvement is measured, not vibes.

Build the harness first. The pipeline is the easy part.

Back to the lab