RAG Pipeline in Production: The Engineering Checklist Before You Ship

The retrieval-augmented generation demo always works. You paste a question, the system fetches the right document, the model answers cleanly, and the room nods. Then you ship it to a few hundred real users querying a real corpus, and it starts inventing policy, citing the wrong contract clause, or taking eight seconds to respond. The failure is rarely the model. It is the engineering around the model.

The canonical survey here is Retrieval-Augmented Generation for Large Language Models: A Survey, which maps the RAG stack into retrieval, augmentation, and generation and makes one thing clear: most of the levers that decide whether your feature is trustworthy sit in the plumbing, not the prompt. Enterprise teams tend to invest heavily in the model choice and almost nothing in the three layers that actually break in production.

Finding 1: Context stuffing is not a chunking strategy

The most common production failure is dumping whole documents — or naively fixed-size splits — into the context window and hoping the model reads them. The survey is explicit that retrieval quality dominates end-to-end quality: if the retrieved chunks don't contain the answer, or contain it buried among irrelevant text, no amount of prompt engineering recovers it. Long contexts also degrade attention to the middle of the window, so a correct chunk ranked fifth can be functionally invisible.

What goes wrong at enterprise scale is subtle. A contract clause split across two chunks loses its meaning. A table divided mid-row becomes noise. A policy document chunked by character count in place of by semantic section returns fragments that read plausibly but answer the wrong question.

The action this week: audit your chunking against your actual document structure, not a default. Chunk by semantic boundary — section, clause, function, ticket — and preserve enough overlap that a boundary-spanning answer survives. Then measure retrieval in isolation: for a sample of real questions, does the top-k set actually contain the answer? If retrieval recall is poor, fix that before you touch the generation prompt. You cannot evaluate an answer the retriever never fetched.

Finding 2: No evaluation framework means you are shipping on vibes

The second failure is having no way to tell whether a change made things better or worse. Teams ship a RAG feature, get a complaint, tweak the prompt, and eyeball three examples to confirm it's fixed — with no idea what they broke elsewhere. The survey stresses that RAG evaluation has to cover both retrieval quality (did we fetch the right context?) and generation quality (did we answer faithfully from that context?), because a good answer from bad context is luck, not reliability.

An evaluation framework for RAG is not exotic. It is a fixed set of representative questions with known-good answers or reference passages, run automatically on every change, scoring retrieval recall, answer faithfulness (is the answer grounded in retrieved text?), and answer relevance. Faithfulness is the one enterprise teams skip and the one that matters most: a fluent answer that isn't supported by the retrieved documents is a hallucination wearing a suit.

The action: build a golden set of 50–100 real questions before launch, drawn from what users will actually ask, with expected sources attached. Run it in CI. Any prompt, chunking, or model change that moves faithfulness or recall down fails the build. This is the same discipline that separates measured quality engineering and test automation from manual spot-checking — you are treating the retrieval layer as a system under test, not a demo.

Finding 3: Latency has a budget, and retrieval spends it fast

The third failure is discovered only under load. A RAG request is a chain: embed the query, search the vector store, rerank, assemble context, call the model, sometimes call the model again. Each hop adds latency, and rerankers and multi-step retrieval — both of which improve answer quality — are exactly the components that add the most. Teams optimise for accuracy in a notebook, then find their p95 latency is unacceptable when real traffic hits.

The survey documents the accuracy gains from reranking and iterative retrieval, but those gains are not free in a user-facing feature. If your product promises a conversational response and the pipeline takes six seconds at the tail, users abandon it regardless of how good the answer is.

The action: set a latency budget per stage before you build, the way you'd set a performance budget for a page load. Decide what the tail latency target is for the whole request, then allocate it across embedding, search, reranking, and generation. Measure each stage in production, both in aggregate. If reranking buys you two accuracy points but blows the budget, that's a product trade-off a human should make deliberately — not a surprise you discover from a support ticket.

The twelfth question: who owns this layer?

Run the checklist before you ship — chunking matched to document structure, retrieval recall measured in isolation, a golden-set evaluation framework in CI, faithfulness scored not assumed, a latency budget per stage, and behaviour under concurrent load tested. But the final question is organisational, not technical: who owns the retrieval layer once it's live?

A RAG feature is not fire-and-forget. Your corpus changes, users ask new kinds of questions, the underlying model version shifts under you, and retrieval quality drifts. Without a named owner and a running evaluation framework, drift is invisible until it's a complaint. Many teams ship the feature and dissolve the working group, leaving nobody watching faithfulness decline as the document set grows.

How a senior pod approaches this

At Anystack, retrieval over a client's own documentation is something the pod uses inside its own delivery — drafting stories from specs, grounding change rationale in the codebase — so the failure modes above are ones we operate against directly, both theorise about. When a client team is standing up a production RAG feature, a senior engineering pod approaches it the same way it approaches any system that has to hold up under load: define the evaluation framework first, measure retrieval and faithfulness separately, set the latency budget explicitly, and put the whole thing under continuous test so regressions are caught in CI in place of in production.

The mechanism that makes this credible is the same one the pod applies everywhere — test-effectiveness measurement and adversarial review, so what reaches production is evidenced against your bar in place of demonstrated once and hoped over. For a RAG layer, that means the golden set is adversarial by design: it includes the questions most likely to trigger hallucination, the documents most likely to be mis-chunked, and the query volumes most likely to breach the latency budget. What ships is what survived that scrutiny.

The pattern across all three findings is that RAG is an engineering problem dressed as an AI problem. Chunking is a data-modelling decision. Evaluation is a test discipline. Latency is a systems budget. None of them is solved by picking a better model, and all of them are solved by the same senior engineering judgement you'd apply to any production system. Ship the checklist, name the owner, and keep the evaluation pipeline running.

Three things to do this week: build a golden set of 50–100 real questions with expected sources and run it before your next change; measure retrieval recall in isolation from generation so you know which layer is failing; and write down a per-stage latency budget so reranking and multi-step retrieval are deliberate choices in place of tail-latency surprises.