RAG in Production: Lessons from Real Deployments

Review update · 17 October 2026. Removed unsupported performance and prevalence claims, qualified chunking/reranking advice and added access-control and context-budget guidance. No new deployment trial is claimed.

The basic RAG architecture is simple enough to prototype in an afternoon. Building one that works reliably across a realistic distribution of queries, on real-world document collections, is a substantially different problem.

The Prototype-to-Production Gap

A typical RAG tutorial covers: load a document, split into chunks, embed with a hosted model, store in a vector database, retrieve top-k chunks at query time, pass to a language model with a prompt. This works — for the tutorial document, on the tutorial queries, it works well.

The gap appears when you move to large, heterogeneous document collections; real user queries that are ambiguous, poorly specified, or outside the document scope; production load with latency requirements; and documents that change over time.

Retrieval Failure: Diagnose It Separately

Poor retrieval can deprive the generator of the evidence it needs. But parsing, permissions, stale documents, generation and evaluation can also fail. Diagnose each stage rather than assert that most failures share one cause without workload evidence.

Compare chunking strategies on your documents

Fixed-size character or token chunking is the default because it is simple. It can perform poorly when the relevant information spans a chunk boundary, when a chunk contains information from multiple unrelated topics, or when document structure (headings, tables, code blocks) is lost by naive splitting.

Semantic chunking splits at semantically coherent boundaries rather than fixed sizes. It adds processing cost and should be compared with simpler baselines on your documents.

Hierarchical chunking stores both fine-grained chunks (for precision) and their parent sections (for context). Retrieve at the fine level, return the parent for generation. Known as the "small-to-big" retrieval pattern.

Document-aware chunking respects document structure — headings, sections, paragraphs, code blocks, tables. Preserving that structure is worth testing, especially for tables and code, but it is not a universal performance guarantee. See Microsoft’s chunking guidance.

Query-document semantic mismatch

There is frequently a semantic gap between how users phrase queries and how relevant information is expressed in documents. A user asking "how do I cancel my subscription?" needs the chunk containing "to terminate your account, navigate to..." — the embeddings may not be close enough.

HyDE (Hypothetical Document Embeddings) generates a hypothetical answer to the query, embeds that, and retrieves against it. The hypothetical answer lives in a similar semantic space to the actual documents. It adds a generation step and can help on some query sets; compare it with direct query retrieval, including failure cases.

Hybrid search combines dense embedding retrieval with sparse BM25 retrieval. Dense handles semantic similarity; sparse handles exact term matching. Reciprocal Rank Fusion (RRF) is a simple and effective way to combine results. Compare it with dense-only and keyword-only retrieval on a labelled set; exact identifiers and domain terms can make lexical matching particularly useful.

The Reranker as a Second-Stage Filter

Vector search can use approximate nearest-neighbour indexing; exact search is also possible, depending on the system. A cross-encoder reranker takes a query-document pair and produces a direct relevance score. It scores candidate pairs and adds work proportional to the candidates evaluated; those pairs can be batched rather than requiring a separate network call each. Whether it improves relevance enough to justify the cost needs measurement.

A candidate set such as 20–50 chunks is a starting point for an experiment, not a production default. Tune both retrieval and final context sizes. Reranking latency depends on the model, candidate lengths, batching, hardware and service; the former blanket “under 100ms” estimate has been removed. Microsoft’s retrieval guide discusses these trade-offs.

Context Window Management

The temptation is to stuff as many retrieved chunks as possible into the prompt. This is usually wrong.

Position sensitivity. The Lost in the Middle study found that performance on its document-question and key-retrieval tasks could depend on where relevant information appeared. This is evidence from the tested models and tasks, not a fixed 30–70% failure zone or proof that every current model behaves identically. Test your deployed model and document lengths.

Context contamination. Irrelevant retrieved content does not just add noise — it can actively degrade model output by providing plausible but wrong information that the model may incorporate.

Practical approach: use a reranker to limit context to the genuinely relevant chunks. Compare context selections using evidence coverage and answer quality; no fixed chunk count wins for every question.

Document Freshness and Index Management

RAG systems serving changing document collections need mechanisms for deletion tracking (when documents are removed, their vectors must be deleted or marked stale), incremental updates (new and changed documents must be re-embedded without full re-indexing), and versioning in some applications.

Most vector databases support soft deletes and metadata filtering. Build document lifecycle management into your architecture from the start — retrofitting it is painful.

Evaluation: The Critical Missing Piece

A useful evaluation separates retrieval from answer generation. This article does not establish how widely production systems are evaluated. A minimum viable evaluation framework:

Retrieval evaluation — separate from generation. Does the retrieval step return the correct documents? Measure precision and recall against a labelled ground truth set. A prompt cannot recover a missing document reliably; first check whether the evidence reached the model.

Faithfulness — is the generated response supported by the retrieved context? The model should not be hallucinating information not present in the retrieved chunks.

Answer quality — does the response correctly answer the question given the context? Use human-labelled examples where feasible. Model-based grading can assist, but calibrate it against human judgments and inspect disagreements.

Failure mode coverage — specifically test questions outside the document scope (the system should indicate it does not know), ambiguous queries, queries requiring multi-document synthesis, and adversarial queries.

Permissions are part of retrieval quality

Filter documents using the requesting user’s authorised access before sending retrieved text to the generator. A request in the prompt to keep secrets is not an access-control boundary. Microsoft documents a security-filter pattern; implement equivalent controls in the stack you use. Revalidate access after updates or revocation and treat retrieved instructions as untrusted content.

What to measure before release

Record evidence recall, unsupported answers, useful refusals, permission leaks, document freshness and latency. Include ambiguous questions, exact identifiers, tables, deleted documents and questions outside the collection. Keep the original evidence alongside the answer so a reviewer can check it.

This is a literature-based engineering review, not a report of new production trials. There is no universal winning chunk size, reranker or context length. Start with a simple baseline and retain complexity only when your evaluation justifies it.