ENGINEERING·AI·8 MIN READ

{{ shareLabel }} Print {{ saveLabel }}

Every week we talk to a team that has decided to build a "RAG-powered assistant." Sometimes they mean a genuinely useful product. More often, they mean they have a document corpus, a vector database subscription, and a vague sense that connecting them to an LLM will produce something valuable. It might. It frequently doesn't — not because RAG is wrong for the job, but because the teams implementing it have skipped the question that determines whether it actually works: what does "works" mean, and how will you measure it?

RAG is a pattern, not a product. Understanding it precisely — including where it fails — is the difference between shipping a feature users trust and shipping a demo that erodes confidence every time it gets something wrong.

What RAG actually is — and where it breaks

At its core, RAG is this: take a user's query, find the most relevant chunks from a document corpus, include them in the prompt context, and let the model synthesize an answer. The model stops hallucinating facts from training data because you've supplied real facts to work with. That's the theory. In practice, every step in that chain has failure modes that compound.

Chunking. How you divide documents into retrievable pieces determines what the model can work with. Chunk too small and you lose context — a sentence that makes sense in paragraph three means nothing alone. Chunk too large and you waste the context window on irrelevant text surrounding the relevant bit. There is no universal right answer. The right chunking strategy depends on your document structure, your query patterns, and how your users phrase questions.

Retrieval quality. Vector similarity is a proxy for semantic relevance, not a measurement of it. A query about "invoice payment terms" may retrieve a document about "invoice formatting guidelines" because the embeddings happen to be close. Keyword search misses synonyms. The best production systems combine both — hybrid retrieval — and then apply a reranker to surface the most actually-relevant chunks before they reach the context window.

Context assembly. Stuffing the top five retrieved chunks into a prompt is not context assembly; it's context dumping. Important information buried in chunk four, surrounded by marginally relevant material from chunks one through three, receives less attention from the model than its importance warrants. This is not a bug in the model — it's a consequence of how attention distributes across long contexts, and it's a design problem you can address.

The trust problem is not technical

The reason most AI features erode trust is not that the model gets things wrong. Models get things wrong all the time in applications people still use. The reason trust breaks is that the model gets things wrong confidently, with no visible reasoning and no way for the user to detect the error at the moment it happens.

When an AI assistant confidently cites a policy that was updated six months ago — and the user discovers this only after acting on it — you haven't just damaged trust in a feature. You've damaged trust in the judgment of whoever shipped the feature without making its limitations visible. That's an organizational credibility problem, not just a product bug.

"Users will tolerate errors. What they won't tolerate is an error they couldn't predict, couldn't detect in the moment, and can't trace afterward."

The design solution is almost never a better model. It is better transparency: show the source document, surface the retrieved chunk, let the user verify the claim. When an AI feature makes its reasoning visible, errors become catchable and correctable rather than silent and compounding. This is not a nice-to-have; it's the architectural decision that separates trustworthy AI features from impressive-but-fragile ones.

The evaluation layer is the real product

Here is the thing that separates teams shipping trustworthy AI features from teams shipping polished demos: the evaluation harness. An eval set is a collection of queries with known correct answers, scored automatically against model outputs. It sounds simple. Most teams don't build it until something goes badly wrong in production.

The evaluation layer is where domain knowledge becomes code. An insurance specialist can tell you which retrieved document should rank first for a coverage question. A legal team can tell you which answer is complete and which is dangerously incomplete. None of that expertise lives in the model — it lives in your eval set. And your eval set is the thing that makes your RAG system yours rather than a generic demo anyone could replicate.

Build the eval set before you optimize the retrieval pipeline. Run it before every deployment. Treat a regression in eval scores with the same urgency as a production incident. This is the engineering discipline that makes AI features get better over time rather than quietly degrading.

When not to use RAG

RAG is the right pattern when your use case depends on specific, current, or proprietary information the base model doesn't have — your documentation, your product catalog, your internal knowledge base. If your users need answers grounded in your data, RAG is almost certainly the right architecture.

It is not the right pattern for everything. Classification tasks, structured data extraction, and code generation often don't benefit from retrieval and add complexity without adding accuracy. If query patterns are narrow and predictable, fine-tuning on a curated dataset may produce more reliable results at lower latency. The question is never "should we use RAG?" but "what architecture best fits the query patterns, the data shape, and the failure tolerance of this specific feature?" RAG is a powerful tool. Reaching for it as a default is a shortcut that usually costs more than it saves.