RAG vs Fine-Tuning: When to Use Which

<!-- DRAFT: review by Aleksei -->

The question arrives in nearly every project that puts a language model in front of company data: should we retrieve the data at query time, or train it into the weights? The two options get discussed as rivals, and teams often pick one and defend it. They solve different problems, and the reason to choose between them is usually not accuracy — it is how often the underlying knowledge changes, and whether the gap you are closing is what the model knows or how the model behaves.

This guide sets out the actual decision criteria, the failure mode each approach has, and the cases where the honest answer is "both" or "neither".

The one distinction that decides most cases

Retrieval-augmented generation (RAG) fetches relevant documents at inference time and places them in the prompt. The model's weights never change; the knowledge lives outside the model, in an index you control.

Fine-tuning continues training on your examples, adjusting weights. The result is baked in: no retrieval step, no extra context, and no way to update a fact without training again.

That difference produces a rule that resolves the majority of real decisions:

If the knowledge changes, retrieve it. If the behaviour needs to change, train it.

A support assistant that must cite the current refund policy needs RAG — the policy will change next quarter, and a fine-tuned model would confidently quote the old one. A model that must always answer in a specific structured format, or adopt a domain's register, or stop hedging on every sentence, needs fine-tuning — no amount of retrieved context reliably fixes a behaviour the base model is disposed toward.

What each one is actually good at

RAG handles changing, attributable, high-volume knowledge

The knowledge base updates independently of the model. Add a document and it is usable immediately, with no training run. Remove one and it stops influencing answers — which matters more than it sounds, because deleting a fact from a fine-tuned model is not a supported operation.

RAG also gives you provenance. The retrieved chunks are the evidence, so the system can cite sources and a reviewer can check whether the answer follows from them. In regulated settings this is frequently the deciding factor on its own.

The corpus can also be far larger than any context window or training budget, because only the top matches are ever loaded.

Fine-tuning handles behaviour, format, and cost per call

Fine-tuning changes the model's dispositions: tone, output structure, how it handles ambiguity, which of several valid answers it prefers. These are properties of the mapping from input to output, and prompt instructions steer them only unreliably.

It also removes the retrieved context from every request. If you are serving high volume and each RAG call carries several thousand tokens of retrieved text, a fine-tuned model that needs none of it is meaningfully cheaper and faster — and that gap compounds at scale.

Finally, fine-tuning teaches genuinely new capability: an internal query language, an unusual annotation scheme, a task the base model has simply not seen.

The failure mode of each

Both approaches fail in ways that are easy to misdiagnose as "the model is bad".

RAG fails at retrieval, not generation. If the right chunk is not in the top-k, the model cannot use it, and it will usually produce a fluent answer from whatever it did receive. The symptom looks like a hallucination; the cause is upstream. Before you touch the model, check whether the correct document was actually retrieved. In practice most RAG debugging is retrieval debugging: chunk boundaries splitting a fact in half, embeddings that miss because the question and the document use different vocabulary, a top-k too small for questions needing several sources.

RAG also struggles with questions that require aggregating across many documents. "What is our refund policy?" retrieves well. "How many customers churned last quarter and why?" does not, because the answer is not in any single chunk.

Fine-tuning fails silently and expensively. A fine-tuned fact cannot be updated or removed without another training run, so a stale model keeps asserting the old answer with the same confidence as the new one. Training on a narrow distribution can also degrade unrelated capability — the model gets better at your task and worse at reasoning it previously handled, which you only notice if you kept a broad evaluation set.

And fine-tuning on facts is a weak way to install knowledge. Weights encode statistical tendencies, not a lookup table. A fact seen a handful of times in training data is not reliably retrievable; the same fact placed in the context window is right there.

When to use both

Production systems frequently need both, because the two axes are independent. A common shape: fine-tune for form, retrieve for content.

A clinical documentation assistant might be fine-tuned so its output always follows the required note structure and clinical register — behaviour that must be consistent and does not change month to month — while retrieving the current drug interaction data, which changes constantly and must be citable. Neither approach covers both requirements alone.

The order matters. Establish retrieval first: it is cheaper, faster to iterate, and it tells you whether your problem was ever a knowledge problem. Only fine-tune once you can point at behaviour that retrieval demonstrably does not fix.

When the answer is neither

Two cases are worth naming, because both get solved with expensive machinery that was not required.

Prompting may be enough. Before either approach, check whether a better prompt with a few examples closes the gap. This is the cheapest experiment available and it succeeds more often than the amount of RAG tooling in the ecosystem would suggest.

The context window may be enough. If the entire corpus is a few dozen pages and it fits in context, retrieval infrastructure buys you complexity and a new failure mode for no benefit. RAG earns its cost when the corpus exceeds what you can pass in — not before.

A decision procedure

Work through these in order and stop at the first that applies.

1. Does the knowledge change, or must answers be attributable? → RAG.
2. Does the whole corpus fit comfortably in context? → Skip retrieval, put it in the prompt.
3. Is the gap in behaviour, format, or register rather than facts? → Fine-tuning.
4. Is per-call cost or latency the binding constraint at high volume? → Fine-tuning, to drop the retrieved context.
5. Both a behaviour gap and changing knowledge? → Both, retrieval first.
6. None of the above clearly? → Better prompting, and measure before building anything.

The explorer above shows the step most RAG failures actually occur at. Retrieval matches on embedding proximity, not meaning: try a question phrased in different vocabulary from the target text and watch the nearest neighbours change. That gap between "semantically related to a human" and "close in embedding space" is where the right chunk quietly fails to make the top-k.

What to measure

Whichever you choose, evaluate the component that can fail rather than the system as a whole.

For RAG, measure retrieval separately from generation. Build a set of questions with known correct source documents and check how often the right one lands in the top-k. If retrieval recall is 70%, the ceiling on the whole system is 70%, and no prompt engineering raises it.

For fine-tuning, keep a held-out set from before training and a broad set covering capability you are not targeting. The first tells you whether it worked; the second tells you what it cost.