Production RAG: Why Vector Search Alone Falls Short

Production RAG is becoming a retrieval-architecture problem rather than a simple vector-database integration. In a Towards AI article published on August 25, 2026, Dave R. argues that a prototype can ingest documents, create embeddings, retrieve chunks, and call a language model, while a production system must also handle unpredictable queries, growing corpora, multi-source answers, citations, and measurable quality.

Definition: Production RAG is a retrieval-and-generation system that prepares trustworthy source context, selects evidence through the right retrieval path, and measures whether the final answer remains grounded.

Example: A question comparing an old policy with a current policy may require document identification, several targeted searches, reranking, and evidence synthesis rather than one nearest-neighbour lookup.

Key takeaway: Vector search is one retrieval signal inside production RAG, not the complete production architecture.

Business impact: Teams can reduce retrieval blind spots by matching search methods to query types and by measuring retrieval, groundedness, and answer relevance separately.

Why does production RAG start with reliable context?

Production RAG succeeds or fails on the quality of the context reaching the language model. The source frames this as a retrieval problem because private reports, contracts, support tickets, policies, and project documents are useful only when an application can find the relevant passages and place them in the model's context. For an operator, the practical starting point is to define the authoritative sources and the evidence path before tuning the model.

The source distinguishes three ways an application can access private or project-specific information: a context window for small, short-lived content; project or session retrieval for a smaller working set; and RAG for indexing a larger corpus and retrieving selected pieces per question. That distinction matters because the retrieval design should follow the size, lifetime, and scope of the data rather than forcing every task into a vector index.

What ingestion work does production RAG require?

Production RAG treats ingestion as a pipeline, not a single embedding call. The source describes parsing, normalization, chunking, embedding, indexing, metadata extraction, and relationship extraction as separate responsibilities that prepare documents for later retrieval. Teams building a RAG system should preserve enough source metadata to trace every returned chunk back to its document and location.

Chunking is a retrieval decision because the size and boundaries of a chunk determine what the embedding represents. The source warns that a whole long document can be too broad for precise retrieval, while a poorly chosen small chunk can lose the surrounding condition or exception that gives a passage its meaning. The practical takeaway is to choose chunking and overlap against real document structures and questions, then keep source identity attached to every chunk.

The existing RAG explainer covers the basic ingestion, retrieval, augmentation, and generation flow. This story extends that baseline by asking what must change when a corpus grows, questions become multi-step, and operators need evidence that a retrieval change improved the system.

Why can one vector query miss the right evidence?

Vector search is useful for semantic similarity, but vector search is not equally precise for every production query. The source contrasts semantic retrieval with keyword matching for exact names, dates, codes, and acronyms, and adds graph retrieval when the answer depends on relationships between entities. Teams should therefore treat retrieval as a combination of signals and choose the combination from the query mix. Background: Siebel 26.6 adds RAG search for repeat support tickets.

Hybrid search combines keyword precision with vector recall by running ranked retrieval methods together and fusing their result lists. The source describes Reciprocal Rank Fusion, or RRF, as the mechanism used by Azure AI Search to merge parallel ranked results, while a semantic reranker can then rescore the candidate set against the question. The operational takeaway is to use hybrid retrieval when both lexical and semantic evidence matter, then rerank before generation instead of sending every candidate directly to the model.

Graph retrieval adds a different signal because graph retrieval follows entities and relationships rather than only comparing passage meaning. The source presents graph retrieval as useful for questions whose answers are distributed across several documents, while the production RAG reference should still begin with the simplest retrieval pattern that answers the current question. Add graph structure when relationships are central to the information problem, not merely because graph-based retrieval is available.

When do complex RAG questions need query planning?

Multi-part questions need query planning when one fixed search cannot gather all required evidence. The source uses a policy-comparison example that requires finding two documents, retrieving relevant sections from each, comparing the evidence, and synthesizing a response. A production RAG system facing similar requests should decompose the question into focused subqueries before generation.

Agentic retrieval moves planning into the retrieval workflow by allowing an agent to inspect the request, create subqueries, use conversation context, run searches in parallel, rerank results, and combine evidence. The source presents this as an architectural change from one fixed lookup to a workflow that decides how much evidence the question requires. Operators should introduce query planning when multi-step requests create measurable retrieval gaps, while keeping simpler questions on a simpler path.

How should production RAG measure quality?

Production RAG needs separate quality signals because a fluent answer can hide weak retrieval. The source identifies three questions: whether retrieval returned relevant chunks, whether the answer is grounded in those chunks, and whether the answer addresses the user's request. Teams should track these dimensions independently so a prompt change is not mistaken for a retrieval improvement.

Retrieval quality measures the evidence selected for the query, groundedness measures whether the generated claims are supported by that evidence, and answer relevance measures whether the response solves the user's task. The source connects these dimensions to Microsoft Foundry evaluators and recommends continuous evaluation and observability rather than occasional manual spot checks. The takeaway is to build a repeatable test set and alert on regressions across the retrieval and generation layers.

The source also describes a feedback loop that evaluates changes to ingestion, ranking, prompts, and data. That loop matters because a system can improve one metric while damaging another: better retrieval recall can add noise, a grounded answer can still be irrelevant, and a relevant answer can still contain unsupported claims. Production RAG teams should keep the metrics separate and inspect the evidence behind surprising results.

What does the production RAG architecture look like?

A production RAG architecture separates an offline ingestion flow, an online retrieval-and-generation flow, and a feedback loop. The source's offline flow parses documents, chunks them, creates embeddings, enriches metadata or entity relationships, and writes retrieval indexes; the online flow plans the query, retrieves candidates, merges and reranks them, and passes selected evidence to the model. Teams can use this separation to change ingestion without silently changing generation behaviour.

The production RAG architecture carries more responsibilities than a vector-only demo: document parsing, lexical and vector indexes, graph relationships where needed, ranking, model calls, citations, evaluation, and the user interface. The source's point is not that every deployment needs every component; it is that the responsibilities still exist somewhere, whether a team composes them from services or uses a managed platform.

The source's implementation example uses financial statements in a .NET and Blazor application. It describes Blazor Server with Telerik UI for Blazor, the Nuclia .NET SDK, and a RAG service handling ingestion, hybrid and graph retrieval, reranking, and evaluation. The example is useful because financial documents combine long text, exact figures, repeated terminology, and questions where a wrong number is easy to detect.

Why do citations and structured output matter?

Citations make a RAG answer inspectable because the user can trace a statement back to the document and chunk that supported it. In the financial-statement example, the source says the system returns a figure together with its source, making the citation part of the result rather than an afterthought. Applications that present evidence alongside generated text should preserve source identity through retrieval, generation, and the final interface.

Structured output removes a fragile parsing step when an application needs to render retrieved facts in a UI. The source describes a C# response type called ChartAugmentedAnswer, passed through an AskAsync flow so the SDK can turn the type into a JSON schema, return schema-shaped JSON, and deserialize it into a typed object for a chart. Teams should use structured output when the answer must drive a component or validation rule, while still retaining the citations that support the values.

A related local hybrid RAG project shows why retrieval design also affects context and cost. The production lesson is the same: retrieval is an application layer that must be measured for evidence quality, not treated as a decorative add-on around a model call.

Which retrieval pattern should teams choose first?

A small temporary source may fit a context window, while a larger corpus with straightforward factual questions may fit classic RAG with careful chunking, keyword and vector search, citations, and evaluation. The source recommends starting with the smallest retrieval pattern that solves the current problem and adding complexity only when application evidence shows it is needed.

Graph retrieval is most valuable when relationships between entities are part of the question, while agentic query planning is more useful when requests require several searches or synthesis steps. The source also distinguishes managed platforms from self-composed services: platforms can reduce retrieval plumbing, while self-managed components provide more control over indexing, orchestration, ranking, and integration. The practical decision is to choose which layer the team needs to own and which failure modes it can operate.

Evaluation should not be postponed until after users depend on the answers. The source's progression is straightforward: build a clean ingestion pipeline, preserve citations and metadata, combine retrieval methods when one signal is insufficient, rerank before generation, plan complex queries, and measure retrieval quality, groundedness, and relevance. That sequence turns production RAG from a demo into a system that can be tested and improved.

What changes when RAG moves beyond the demo?

The source's central argument is that a RAG prototype proves retrieval can work, while a production application must prove that it can keep finding the right evidence as data, questions, and users change. That change requires more than a vector database: it requires ingestion discipline, multiple retrieval options, ranking, planning, citations, structured responses, and continuous evaluation.

For operators, the next step is not to add every advanced retrieval feature at once. Start with a narrow workflow and authoritative sources, measure the retrieval and answer failure modes, and add hybrid search, graph retrieval, reranking, or query planning when the evidence justifies each layer. Production RAG becomes easier to explain and safer to improve when every layer has a defined job and a measurable outcome.