Designing a RAG system that actually works

RAG quality is dominated by retrieval, not by the LLM. Most production RAG systems that underperform have the same root cause: the team treated it as a generation problem when it is actually a retrieval problem with a generation step at the end.

This essay assumes RAG is the right tool for the problem (see When would you fine-tune vs prompt vs RAG? for that decision). What follows is the architecture and operational discipline that separate “we have a RAG system” from “our RAG system works.”

The mental model

A RAG system is a small information-retrieval system bolted to an LLM. The LLM generates answers; everything else is IR.

IR is a 60-year-old field with hard-won lessons that LLM teams routinely re-discover:

Retrieval quality dominates final quality. If the right document isn’t in the top-K, no amount of prompt engineering recovers it.
Embeddings are not a panacea. Hybrid retrieval (lexical + dense) almost always beats either alone.
Reranking is the cheapest big quality win available. Most teams skip it.
Chunking strategy has more impact on quality than embedding model choice. Most teams pick chunk size by intuition and never tune it.

The reference architecture

A production RAG system in 2026 has roughly seven components:

Document ingestion and chunking. Parse source documents into retrievable units.
Embedding and indexing. Compute vectors for each chunk; store them in a vector store.
Lexical index. A traditional keyword index (BM25, etc.) over the same chunks.
Query understanding. Reformulate or expand the user’s query before retrieval.
Hybrid retrieval. Run both dense (vector) and lexical retrieval; combine.
Reranking. Re-order top ~50-200 retrieval candidates down to top ~5-10 for the LLM.
Generation. The LLM, with the retrieved chunks in context, produces the final answer with citations.

Plus the operational layer: an eval pipeline (see LLM Evals), monitoring, and a feedback loop from production traffic back into the eval set.

Skipping components 4-6 is the single most common failure.

Learning objective

Trace how hybrid retrieval protects recall before a query-aware reranker selects a small, precise context.

Preserve both evidence passages before reranking their relevance

Derivation

follow the same query down both first-stage lists. Lexical search preserves the exact error-code passage A; dense search preserves the paraphrased recovery passage B. Fuse those ranks to keep both relevant candidates, then let the slower query-passage reranker choose their order and discard distractions before generation. The first stage protects recall; the second buys precision. This is an original synthetic example informed by Karpukhin et al.'s dense passage retrieval, Cormack et al.'s reciprocal rank fusion, and Nogueira and Cho's BERT reranker.

Check: Why must both passages survive broad retrieval before the reranker can choose precise context?

Read it this way: Both required evidence passages must survive retrieval and filtering. A relevant passage dropped before reranking cannot appear in the final answer context.
Can reranking recover evidence B after the cap removed it?

Component-by-component, with the things that actually matter

Chunking

Fixed-size chunks are a valid baseline, not a universal mistake. Compare what each boundary preserves or separates under the corpus, query distribution, and token budget.

Better approaches, in rough order of sophistication:

Semantic chunking: split on paragraph or section boundaries, then merge to a target token count.
Heading-aware chunking: retain parent titles and section context when these clarify a passage; measure whether repeated metadata helps retrieval or consumes useful context.
Sliding-window with overlap: 256-token chunks with 50-token overlap, so context near boundaries isn’t lost.
Hierarchical chunking: keep chunks at multiple granularities (paragraph + section + document) and retrieve at the right level for each query.

These are candidate boundary policies, not a guaranteed quality ordering. Measure whether the required authorized evidence survives chunking, retrieval, selection, and generation, as in the RAG mechanism.

Tune chunk size empirically against your eval set. The right size varies more by domain than people expect; retuning it is often a high-impact, low-effort change.

Embedding model

Compare embedding candidates on representative queries, languages, access-controlled corpora, and serving constraints. A public benchmark ranking does not establish either small model differences or a larger guaranteed hybrid-retrieval gain for this corpus.

Two non-obvious points:

Domain-specific embeddings matter for specialized text. Out-of-the-box embeddings underperform on legal, medical, or code search. Fine-tuning embeddings on your domain (with synthetic queries) typically helps materially.
Embedding model choice locks you in. Switching means re-embedding your entire corpus. Pick something you’ll commit to; don’t churn weekly.

Lexical retrieval (don’t skip this)

The most common architecture mistake in RAG is using only dense retrieval. Lexical retrieval (BM25 in particular) wins on:

Exact-match queries (product codes, error messages, named entities)
Out-of-distribution terms not well represented in embedding training data
Acronyms, abbreviations, and domain jargon
Numeric references (“CVE-2023-1234”)

Hybrid retrieval combines different candidate signals, for example through reciprocal rank fusion. Evaluate authorized evidence recall, latency, and failure slices against the lexical-only and dense-only baselines. Additional stages have engineering and resource costs; they are not automatically the right first release.

Query understanding

The user’s query is rarely the right query for retrieval. A few transformations that help:

Query expansion / HyDE: have an LLM generate a hypothetical answer to the query, then embed the hypothetical answer (not the query) and search. Counterintuitively effective; works because the answer is closer in embedding space to actual answer documents than the question is.
Query decomposition: for complex queries (“how does our refund policy interact with the EU consumer protection law”), break into sub-queries, retrieve for each, combine. Especially important for multi-document RAG.
Query rewriting: rephrase ambiguous queries into clearer ones. Often as simple as “given this conversation history, rewrite the user’s last message into a self-contained query.”

Query transformations can improve a retrieval request or change its intent. Retain an interpretable original-query baseline and test ambiguous, adversarial, and multi-part requests before claiming robustness.

Reranking (the highest-leverage step)

After retrieval returns the top ~50-200 candidates, a reranker scores each (query, candidate) pair more carefully and re-orders.

Why it works: a bi-encoder embedding model has to encode query and document independently, so it can only model coarse similarity. A cross-encoder reranker sees both at once and can model fine-grained relevance, which terms matched, which paraphrases hold, whether the candidate actually answers the question.

Whether reranking is the highest-value change depends on the diagnosed failure and measured costs. A reranker can reorder the retrieved set but cannot recover a required passage that never entered it. If the failure is missing evidence, improve recall or the retrieval boundary; if candidates are adequate, test bounded reranking before adopting it.

Generation

Generation remains a separate failure boundary even when retrieval is correct. Test whether each consequential claim is supported, current, and permitted for release; a resolving citation alone proves none of those. Useful candidate controls include:

Cite-by-passage prompting: instruct the model to cite which retrieved passage each claim comes from, with passage IDs. Improves trustworthiness and makes hallucinations easier to catch.
Grounding instructions: explicitly tell the model “if the answer is not in the provided documents, say so.” Without this, models will hallucinate confidently.
Context window management: don’t dump 50 chunks into context. Top 5-10 (after reranking) usually beats top 50, both in quality (less distraction) and cost.
Structured output when applicable: for many use cases (Q&A with citations, extraction), structured JSON output is strictly better than free-form prose.

Common failure modes

In rough frequency order:

Retrieval is bad and team doesn’t realize it. They evaluate the end-to-end RAG output and try to improve it by changing prompts. They never look at whether the right document was even retrieved. Fix: separately evaluate retrieval (hit rate at K, MRR) before evaluating generation.
Chunk size is wrong. Default 512-token chunks for documents where 200 or 1024 would have been right. Fix: tune empirically.
No reranker. Team uses raw bi-encoder retrieval and wonders why precision is low. Fix: add a reranker. This is usually a one-day improvement worth 10pp.
No lexical retrieval. Team relies entirely on embeddings and fails on exact-match queries. Fix: add BM25, fuse with reciprocal rank fusion.
Eval set is too small or doesn’t reflect real queries. Team has 30 hand-crafted Q&A pairs, none of which look like real user queries. Fix: build the eval set from production traffic samples (see LLM Evals).
Citations don’t actually verify. Model cites passages that don’t support its claims. Fix: add a citation-verification step where you check each citation against the retrieved passage; flag ungrounded claims.
Stale index. Team indexes once, then never re-indexes as documents change. Fix: incremental indexing pipeline from day one.
Chunking strips structure. PDF parsing loses headings, tables, lists, and the chunks become decontextualized text. Fix: use a structure-aware parser; preserve heading hierarchy.

What an interviewer expects you to discuss

If asked to “design a RAG system” in a senior loop, the answer should hit:

The 7-component reference architecture (or a defensible variant).
Hybrid retrieval (dense + lexical), with explanation of why.
A reranker as a separate stage.
Query understanding, especially HyDE or decomposition for hard queries.
An eval framework with separate retrieval and generation metrics.
A specific failure mode you’ve seen and how you fixed it (this is the L6 signal).
Cost and latency considerations, reranking is fast but indexing is expensive; embedding-update strategies for evolving corpora.

If you list “embedding model + vector store + LLM” and stop there, you’ve signaled L4 understanding regardless of how good the embedding model is.

Tools to consider in 2026

Vector store: pgvector for < 10M docs, Qdrant or Weaviate for larger, Vespa if you need both lexical and dense in one system.
Embeddings: BGE-M3 or NVIDIA NV-Embed for open; Cohere v3 or OpenAI text-embedding-3-large for managed.
Reranker: Cohere Rerank, BGE Reranker v2, or a fine-tuned in-house cross-encoder.
Lexical: Elasticsearch / OpenSearch / Tantivy, whatever your team already operates.
Orchestration: don’t use LangChain. Use a thin custom layer or LlamaIndex if you want managed primitives.
Eval: Inspect AI, Promptfoo, or Braintrust for the framework; pytest for the lightweight version.

The choice of vendor matters less than the discipline of running them well. Excellent RAG systems exist on the simplest stacks; the most expensive stacks don’t guarantee quality.

The discipline that separates the good teams

Three habits, in order of importance:

They evaluate retrieval and generation separately. Hit rate at K, MRR, recall at K for retrieval; faithfulness, citation accuracy, answer quality for generation. Both metrics, every release.
They watch the failure tail, not the average. A 95% top-1 hit rate hides a catastrophic 5% where users get the wrong document and a hallucinated answer. The 5% is what loses trust.
They look at outputs. Same as the LLM evals advice: read the actual responses every week. The metrics tell you what changed; the outputs tell you what’s actually happening.

Build these habits before you build anything fancy. The fancy parts are easy; the discipline is hard.