Building RAG systems that hold up in production

A retrieval-augmented generation demo takes an afternoon. You embed some documents, drop them in a vector store, stuff the top five matches into a prompt, and the answers look convincing. The gap between that and a system people trust with real decisions is where most projects quietly stall.

The failure is rarely the model. It is almost always retrieval: the right passage was never fetched, so the model filled the gap with something plausible. If you only evaluate end-to-end answer quality, you cannot tell those two failures apart.

Evaluate retrieval separately from generation

Before measuring answer quality, measure whether the correct source was retrieved at all. Build a fixed set of questions with known supporting passages and track recall at k. This single number tells you whether to spend your next week on chunking and indexing or on prompting and model choice.

• Recall@k — was the supporting passage in the retrieved set?
• Groundedness — is every claim in the answer traceable to a retrieved passage?
• Answer relevance — does the response actually address the question asked?
• Refusal rate — how often does the system correctly decline when context is missing?

That last metric matters more than teams expect. A system that confidently answers everything is worse than one that says it does not know, because it transfers the verification burden back to the reader without telling them.

Chunking is a retrieval decision, not a preprocessing detail

Fixed-size chunks split tables mid-row and separate headings from the text they govern. Structure-aware chunking — respecting document sections, keeping headings attached to their body, preserving table integrity — usually delivers a larger accuracy gain than switching to a bigger model, at a fraction of the cost.

Hybrid retrieval helps for the same reason. Dense vectors capture meaning; keyword search captures exact identifiers, product codes and proper nouns that embeddings routinely blur together. Most production systems need both, combined with a reranking pass over the merged candidate set.

Make the system show its work

If a user cannot check an answer against its source in one click, you have not built a knowledge system. You have built a confident stranger.

Citations are not a UI nicety. They are the mechanism by which a probabilistic system becomes auditable, and they change adoption more than any accuracy improvement. Every claim should link to the passage it came from, and users should be able to disagree with an answer in a way you can capture and learn from.

Plan for the corpus changing underneath you

Documents get superseded, access rules change, and stale content keeps answering questions long after it should. Production retrieval needs incremental re-indexing, document-level permissions enforced at query time rather than in the prompt, and a way to retire content without rebuilding the index from scratch.

None of this is exotic. It is ordinary engineering discipline applied to a probabilistic component — which is exactly the work that separates a system that survives contact with real users from one that demos well and is quietly abandoned two quarters later.