When RAG Is the Wrong Answer

Retrieval augmented generation has become the default response to every document problem. Several cheaper options are worth ruling out first.

Hafeez Baig•Feb 18, 2026•9 min read

Introduction

A team has documents. Someone needs to get answers out of them. Within about ten minutes, the plan is a vector database, an embedding model, a chunking strategy, and a retrieval pipeline. Nobody asked whether the problem actually requires any of that.

This post is not an argument against retrieval augmented generation. RAG is a good technique and there are problems where nothing else will do. The argument is narrower: RAG has become the default, and defaults do not get examined. It carries real operational cost, and a meaningful share of the systems built with it would work better as a search box, a SQL query, or a text file pasted into a prompt.

One definition first. Retrieval augmented generation means: before you ask the language model a question, you fetch some relevant text from your own data and paste that text into the prompt alongside the question. The model then answers using the material you supplied. "Retrieval" is the fetching step. "Augmented generation" is the model answering with the fetched text in front of it.

The usual implementation fetches that text by semantic search. Documents get split into chunks, each chunk is converted into a vector (a list of numbers representing its meaning), and those vectors go into a vector store, a database built to find the vectors closest in meaning to your question. That machinery is what most people mean when they say "we built RAG."

What follows is a ladder of increasing complexity. Stop at the first rung that solves the problem.

1. Plain Search

Start with the most basic question, and be honest about the answer: does the user want to find a document, or ask a question?

These are different needs and they get conflated constantly. Someone looking for "the Q3 vendor contract" does not want a generated paragraph summarising the contract. They want the contract. A search result list with a good filter UI gives them the file in one click. A RAG pipeline gives them a paragraph that may or may not be accurate, and they still have to open the file to be sure.

Full text search is fast, cheap, has no per query model cost, and is predictable, which matters more than people credit. When a keyword search returns nothing, the user understands exactly what happened and adjusts their words. When a retrieval system returns a wrong but fluent answer, the user has no idea anything went wrong.

Reach for search first when:

Users know roughly what they are looking for.
The result they want is the document itself.
Metadata such as date, author, type, or status does most of the narrowing.
Exact terms matter, for example product codes or error codes.

That last point deserves weight. Semantic search is weaker at exact identifiers than plain keyword matching. If your users search for strings like ERR_5521, embeddings are the wrong tool even for the retrieval step.

2. A Database Query

If the answer lives in structured data, query it, do not embed it.

This sounds obvious written down, and it is violated all the time. Teams take rows from a transactions table, turn each row into a sentence, embed it, and load it into a vector store so users can "ask questions about the data." Then they discover that "how many orders shipped late last month" returns whichever rows happened to be semantically nearest, which is not a count. It is an impression.

Aggregation, filtering, joins, and sorting are what relational databases have been optimised for over decades. Nearest neighbour search cannot do them, and pretending otherwise produces confidently wrong numbers.

The right pattern here is usually natural language to SQL: let the model write a query against a well defined schema, run the query, and return real results.

sql

-- The model generates this. The database computes the answer.
SELECT COUNT(*)
FROM orders
WHERE shipped_at > promised_at
  AND order_date >= '2026-01-01'
  AND order_date <  '2026-02-01';

The model is doing translation, not arithmetic. The count comes from the database, so it is correct by construction. This approach has its own failure modes, mostly ambiguous schemas, which is why it works best over a curated view with clear column names rather than raw production tables.

A rule of thumb: if the answer could be a number in a report, it belongs in a query.

3. Put the Documents in the Context Window

The option most often skipped: when the corpus is small and stable, send the whole thing to the model.

Long context models can accept a substantial amount of text in a single prompt. The exact limits move constantly and vary by model, so check that on the day you build rather than treating it as fixed. The point is directional: a corpus that would have been unthinkable to pass whole a few years ago is now often reasonable.

The advantage is not just simplicity. Retrieval is a lossy filter, and skipping it removes an entire class of failure. If the model has every document in front of it, it cannot fail to retrieve the right chunk, because there was no retrieval step to get wrong.

What this deletes from the system:

Component	RAG pipeline	Full context
Chunking logic	Required, needs tuning	None
Embedding model	Required, plus a bill	None
Vector store	Deployed and operated	None
Re-indexing on change	Required	Resend the text
Retrieval quality tuning	Ongoing	Not applicable
Cost driver	Storage, embeddings, queries	Input tokens

The trade is real. You pay in input tokens on every call, and long prompts add latency. Prompt caching helps when the document set is stable. The honest summary is that this rung fails at scale and works well below it.

A good candidate: a company handbook, a product spec, a set of policy documents. Bounded, changes monthly at most, everyone sees the same version.

4. Cache and Precompute

Look at the question distribution before you build anything that answers questions live.

Real usage is rarely uniform. A support tool typically gets the same small set of questions over and over, phrased differently, with a long tail of genuine novelty behind them. If a large share of traffic is a known set of repeated questions, you do not need to retrieve anything at request time. You need an FAQ, generated once and served.

The pattern:

Collect the questions users actually ask.
Cluster them into groups of near duplicates.
Generate an answer for each group, and have a human review it.
Serve those answers directly, matching incoming questions against the known set.
Fall back to something more expensive only for questions that miss.

For the questions it covers, this beats live retrieval on almost every axis. It is faster and cheaper, since there is no model call. Above all it is reviewable: a person has read the answer before a user ever sees it, which is not true of anything generated on demand. In a high stakes domain, that property alone can decide the design.

5. Fine Tuning

Fine tuning is often presented as the alternative to RAG, which sets up the comparison badly. They solve different problems, and the distinction is the most useful thing to hold onto here:

Fine tuning teaches behaviour. Retrieval supplies facts.

Fine tuning means continuing to train a model on your own examples so it internalises a pattern. It works well for:

Style and tone, so output sounds like your organisation without a thousand word prompt.
Format adherence, when you need a specific structure every time.
Classification and routing, where a smaller tuned model can match a larger prompted one at lower cost and latency.

It works poorly for facts. Facts baked into weights cannot be updated without retraining, cannot be cited, and cannot be restricted per user. If your requirement is "the model should answer in our house format," fine tuning is a strong candidate. If it is "the model should know last Tuesday's policy change," it is the wrong tool.

6. When RAG Is Genuinely Right

The fair statement of the other side. RAG is the correct answer when several of these hold at once:

The corpus is large. Too large for any context window, and no filter narrows it enough in advance.
The corpus changes. Documents are added and revised on a timescale no training cycle can follow.
You need citations. Retrieval gives you provenance for free, because you know which chunks you passed in. In some domains this alone is non negotiable.
Access is per user or per tenant. Different people are permitted to see different documents. Retrieval can filter by permission at query time. Weights cannot.
Freshness beats everything. The facts move faster than any offline process.

Large plus changing plus permissioned plus cited is the real home ground for RAG. It is a genuinely hard problem and retrieval is a genuinely good answer to it. I would reach for RAG when at least three of those five apply, and treat one alone as a weak signal.

7. The Hidden Costs

The reason to rule out the cheaper rungs first is that RAG is not a library you import. It is a system you own, permanently.

An ingestion pipeline. Documents arrive as PDFs, scans, spreadsheets, and wiki exports. Parsing them reliably is unglamorous work that never quite finishes.
A chunking strategy. Split too small and you lose context. Too large and retrieval gets imprecise. There is no universally correct setting, only one tuned to your documents.
An embedding bill. Every chunk, every re-index, forever. Changing embedding model means re-embedding everything.
A vector store to operate. Another stateful service to deploy, monitor, back up, and upgrade.
Re-indexing on change. Documents get edited and deleted. Stale vectors serve deleted content back to users, which is worse than a missing answer.
Evaluation. Without a test set you cannot tell whether a change improved retrieval or quietly broke it.
More failure modes. Every stage fails independently. The retriever misses. The chunk lands mid sentence. The right chunk ranks fourth when you take three. The model ignores what you gave it. Each failure produces a fluent, plausible, wrong answer, the hardest kind of bug to notice.

None of these is a reason to avoid RAG. They are reasons to be sure you need it.

8. The Decision Path

Work down and stop at the first match.

text

Does the user want the document itself?
  -> Search with filters.

Is the answer computed from structured data?
  -> SQL, or natural language to SQL over a defined schema.

Is the corpus small and stable?
  -> Put it all in the context window.

Is it the same handful of questions repeatedly?
  -> Precompute answers, review them, serve them.

Is the need style, format, or classification rather than facts?
  -> Fine tune.

Large, changing, permissioned, and needs citations?
  -> RAG. Build it properly.

Most document problems exit this list before the last line.

Conclusion

RAG became the default because it arrived at the moment everyone had documents and a language model, and it fit the problem well enough to stop the search. That is how defaults form. They are rarely wrong so much as unexamined.

The ladder is the useful habit. Ask what the user actually wants, whether the data is structured, how big and how stable the corpus is, and whether the same questions repeat. Each answer either eliminates a rung or confirms it. By the time you reach the top, you either have a much simpler system than you expected, or a clear reason for the complexity you are taking on.

Both outcomes are wins. The failure mode is skipping the ladder entirely and finding out much later that a vector store, an ingestion pipeline, and an evaluation suite were all standing in for a search box.

Build the simplest thing that cannot be talked out of the right answer.