RAG in Production: The Retrieval Problems Nobody Warns You About

A demo RAG pipeline takes an afternoon. Making it answer correctly on a real corpus is a different job, and almost all of the difficulty sits in retrieval.

Hafeez Baig•Feb 4, 2026•11 min read

Introduction

The demo works. You point a script at a folder of documents, split them into pieces, embed the pieces, store them in a vector database, and ask a question. The answer comes back correct and well written. It took an afternoon.

Then you load the real corpus. Ten thousand documents instead of ten, and real users instead of the three questions you invented. The answers get worse in a way that is hard to characterise. Wrong occasionally, confidently, with no signal that anything went astray. The instinct is to blame the model and reach for a bigger one. That is almost always the wrong diagnosis.

First, the definition. RAG stands for Retrieval Augmented Generation. A language model does not know about your private documents, so instead of retraining it, you find the relevant ones at question time and paste them into the prompt. The model answers from that text rather than its memory. The pipeline has two halves: retrieval, which finds the text, and generation, which writes the answer from it. Generation is largely solved and improves every time a better model ships. Retrieval is your problem, and it is where nearly every production failure originates.

1. Chunking Is a Real Decision, Not a Parameter

Documents are too long to embed whole, so they get split into chunks: smaller pieces of text, each stored and searched independently. Most tutorials split every 500 characters and move on. That line sets the ceiling on everything downstream.

Chunks that are too small lose their context. A chunk reading "This limit does not apply to enterprise accounts" is useless alone. Which limit? The embedding carries almost nothing about the subject the sentence modifies.

Chunks that are too large dilute the embedding. An embedding is a single vector, a fixed list of numbers, meant to represent the meaning of the text. Give it three pages spanning four topics and it becomes an average of all four, close to nothing in particular.

Worse is fixed-size splitting cutting through structure, severing a table header from its rows and leaving one chunk of column names and another of unlabelled numbers. Structure aware splitting is the better default: split on the document's own boundaries first, and fall back to size limits only inside those.

python

# Naive: respects nothing but a character count
chunks = [text[i:i+500] for i in range(0, len(text), 500)]

# Structure aware: split on the document's own seams, largest
# first, cutting mid-paragraph only as a last resort.
splitter = RecursiveCharacterTextSplitter(
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
    chunk_size=800,
    chunk_overlap=100,
)

Two additions matter more than the exact size. Overlap repeats a little text between neighbours so a boundary sentence survives intact somewhere. Prepending context, the document title and section heading, gives an isolated fragment something to anchor to.

Stored as: "Billing > Enterprise Plans > Limits: This limit does not apply to enterprise accounts." Now retrievable by someone asking about enterprise billing limits. Before, it was not.

2. Semantic Similarity Is Not Relevance

This is the most important idea here, and the one that surprises people.

Vector search finds text with a similar meaning to the query. It does not find text that answers the query. Those overlap often enough to make a demo work and diverge often enough to break production.

A user asks: "Can I export my data after cancelling?" Vector search returns a paragraph about the export feature and a paragraph about the cancellation flow. Both are highly similar to the question. Both are about the right subject. Neither answers it. The one sentence that does might live in a retention policy that never uses the word "export".

The model receives two topically perfect chunks lacking the answer and does what models do: it produces a fluent, plausible answer anyway. Similarity is a proxy for relevance. A decent proxy, not the same thing, and the gap is where your errors live.

3. The Vocabulary Mismatch Problem

Users do not write like documents. The document says "deprovisioning". The user says "how do I remove someone". The document says "authentication token lifetime". The user says "why do I keep getting logged out".

Embeddings handle some of this, which is their advantage over keyword search. They break down on domain vocabulary, where the link between the user's words and the document's words is a fact about your organisation rather than about English. No general-purpose embedding model knows your internal codename refers to the billing system.

Mitigations, in the order I would try them:

Query rewriting. Use a cheap model to expand the question into two or three phrasings, then pool the results.
A glossary layer mapping internal terms to plain language. Unglamorous, effective, cheap to maintain.
HyDE (Hypothetical Document Embeddings). Have the model draft a fake answer, then search with that. An answer resembles a document more than a question does.

4. Vector Search Cannot Find Exact Strings

Embeddings represent meaning. Identifiers have no meaning. This is a structural mismatch, not a tuning problem.

Search for error code ERR_4021 and a vector index will confidently return ERR_4022 and ERR_4019, because those strings are nearly identical in the ways an embedding model can perceive. The same hits SKUs, version numbers, invoice references, function names, and surnames. Anything where being close is the same as being wrong.

Keyword search is excellent at exactly this and poor at everything in section 3. The two fail in opposite directions, which is why you run both. This is hybrid search: keyword and vector retrieval in parallel, then merge the ranked lists.

python

def reciprocal_rank_fusion(result_lists, k=60):
    """Merge ranked lists. A doc scores well by ranking
    decently in several, not top in exactly one."""
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

merged = reciprocal_rank_fusion([
    bm25_search(query, limit=50),    # exact terms, codes, names
    vector_search(query, limit=50),  # meaning, paraphrase
])

If you make one change to a struggling pipeline, make it this one.

5. Reranking as a Second Pass

Retrieval optimises for speed across millions of chunks, so it uses a cheap comparison: embed the query, find the nearest vectors. That cheapness is what makes it approximate.

A reranker is a slower, more accurate model that reads the query and a candidate chunk together and scores how well that chunk answers that question. It cannot run over your corpus. It runs over the fifty candidates retrieval already narrowed to.

Retrieve wide and cheap, then rerank narrow and expensive. This directly attacks section 2: retrieval finds similar text, and the reranker is the component actually asking whether the text is relevant.

6. The Top-K Problem

Top-k is how many chunks you pass to the model. It looks like a knob. It is a trade-off with no good answer at either extreme.

Set it low and you miss. The answer ranked sixth, you passed three, the model never saw it and answered from what it had. Set it high and you bury the answer in noise. Models attend unevenly across a long context, and the middle is weakest. A correct chunk sitting eleventh out of twenty may as well not have been retrieved. You also pay for every token on every query.

Stop treating k as fixed. Retrieve wide, rerank, then cut on score rather than count: keep chunks above a relevance threshold, up to a ceiling. Some questions deserve two chunks. Some deserve twelve. Few deserve exactly five every time.

7. Stale, Duplicated, and Conflicting Documents

Your corpus is not a clean dataset. It is an accumulation.

Duplication wastes the context window. The same paragraph appears in four documents, so all four fill your top-k with one idea while the answer sits at rank seven. Deduplicate at ingestion by content hash, and near-deduplicate at retrieval.

Staleness and conflict are worse, because retrieval has no concept of truth. Vector search cannot tell that the 2023 policy was superseded in 2025. Both match, both get returned, and the model has no basis for choosing. Sometimes it picks the old one. Sometimes it splices both into a confident answer describing a policy that has never existed.

The embedding carries no authority signal, so you supply it:

Store effective_date, version, status, and source_authority on every chunk.
Filter deprecated content at query time rather than hoping the model notices.
Boost recent and authoritative sources during ranking.
When conflicts reach the context, label chunks with dates and tell the model to prefer the most recent and surface the conflict rather than silently resolve it.

8. Metadata Filtering Is Underrated

Teams reach for better embeddings when a WHERE clause would have done the job.

If a user asks about the current pricing tier for their region, no semantic tuning beats filtering to region = "EU" AND status = "current" before the vector search runs. You have shrunk the haystack, and everything downstream gets easier and cheaper.

The work is mostly at ingestion: extract structured metadata while you still have the document, its path, and its title in hand. That information is expensive to reconstruct later and free to capture now.

9. Multi-Tenant Retrieval

If your system serves multiple customers from one index, retrieval is not only a quality concern. It is a security boundary.

A vector index has no natural sense of ownership. Tenant A's question and Tenant B's document can be extremely similar, which is exactly the condition under which nearest-neighbour search returns the wrong one. The result is one customer's confidential text rendered into another customer's answer, in fluent prose, with no error raised anywhere.

python

# The filter belongs here, applied unconditionally,
# not in whichever caller happens to remember it.
def search(query: str, session: Session, k: int = 50):
    return index.query(
        vector=embed(query),
        filter={"tenant_id": session.tenant_id},  # from auth, never from input
        top_k=k,
    )

Enforce the tenant filter in the data layer, never derive the tenant from the request body, and consider separate namespaces per tenant. Then test it: ask tenant A's question against a corpus seeded with a distinctive tenant B document, and assert that document never appears.

10. The Confident Answer From a Bad Chunk

Every problem above converges here.

Retrieval returns something topically plausible but wrong. The model receives it as context, and the instruction says to answer from the context. So it does, writing a clear, well structured, entirely wrong answer with no hedging, because nothing told it the chunk was bad. The chunk arrived looking exactly like a good chunk.

A model with no information tends to sound uncertain. A model given wrong information sounds completely sure. The system is least trustworthy precisely when it appears most authoritative, and users calibrate their trust on fluency.

Partial defences: give the model an explicit path to say the context does not contain the answer, require citations so a human can check, and refuse below a relevance floor. None of these fix bad retrieval. They make it visible, which is the next best thing.

11. Evaluation Is the Thing Everyone Skips

Every section above describes a tuning decision. Chunk size. Overlap. Hybrid weights. Rerank depth. Top-k.

Without evaluation, all of them are guesses. You change the chunk size, try three questions, they look better, you ship. You have no idea what you broke, because you tested three samples against a corpus of thousands.

The minimum viable set is smaller than people expect: a few dozen real questions with known correct source documents. Not generated questions. Real ones, including the awkward ones. Then measure the halves separately, because they fail separately:

Retrieval: did the correct chunk appear at all, and how highly was it ranked. A mechanical check, no model judgement required.
Generation: given that the correct chunk was retrieved, was the answer right and grounded in it.

If the correct chunk was never retrieved, no prompt engineering will save you. Without that split, every failure looks the same from outside, and every fix is a coin toss.

Failure Modes and Mitigations

Failure mode	What you observe	Mitigation
Chunks too small	Fragments lacking context	Overlap, prepend title and heading
Chunks too large	Relevant chunks never rank	Structure aware splitting, tighter cap
Split through structure	Broken tables, orphaned numbers	Split on document boundaries
Similar but not relevant	On-topic chunks, unanswered question	Add a reranker
Vocabulary mismatch	Users' words find nothing	Query rewriting, glossary, HyDE
Exact IDs not found	Wrong error code or SKU returned	Hybrid search with keyword retrieval
Top-k too low	Answer exists but never passed	Retrieve wide, rerank, cut on score
Top-k too high	Answer present but ignored	Score threshold, dedupe
Duplicated documents	Context filled with one repeated idea	Hash dedupe at ingest
Stale documents	Confidently outdated answers	Freshness metadata, filter deprecated
Conflicting documents	Answers flip between runs	Authority metadata, label dates
Cross-tenant leakage	Another customer's data in an answer	Filter in the data layer from session auth
Confident wrong answer	Fluent, sourceless, incorrect	Citations, score floors, refusal path
No evaluation	Every change is a guess	Measure retrieval and generation apart

Conclusion

When someone says the model is dumb, the useful next question is: what was in its context window.

Most of the time that explains everything. The right chunk was not there. It was ranked eleventh in a wall of noise. It sat alongside a superseded version of itself. It was cut in half by a character counter that has never read a document. The model was not being dumb. It was doing exactly what it was asked with what it was given.

Generation quality improves every time a new model ships. Retrieval quality is yours alone, and no model release will improve it. That asymmetry should decide where the effort goes.

Start with the evaluation set, because otherwise the rest is guesswork. Then hybrid search, then a reranker, then metadata. Chunking last, once you can measure whether a change helped.

The model is rarely the bottleneck. The context you hand it usually is.

RAG in Production: The Retrieval Problems Nobody Warns You About

Introduction

1. Chunking Is a Real Decision, Not a Parameter

2. Semantic Similarity Is Not Relevance

3. The Vocabulary Mismatch Problem

4. Vector Search Cannot Find Exact Strings

5. Reranking as a Second Pass

6. The Top-K Problem

7. Stale, Duplicated, and Conflicting Documents

8. Metadata Filtering Is Underrated

9. Multi-Tenant Retrieval

10. The Confident Answer From a Bad Chunk

11. Evaluation Is the Thing Everyone Skips

Failure Modes and Mitigations

Conclusion

Subscribe to my newsletter