RAG or fine-tuning: which does your AI product actually need?

The decision is easier once you stop treating these as competing options and diagnose the failure you actually have.

If the model produces fluent answers that are factually wrong about your business, that is a knowledge problem. It has never seen your pricing, your policies, or this customer's order history. Fine-tuning is a poor fix, because you would be trying to bake facts into weights that need updating whenever the facts change.

If the model knows the domain but returns prose when you need strict JSON, adopts the wrong register, or drifts on a narrow repetitive task, that is a behaviour problem. Retrieval will not fix it, because nothing is missing from the context — the model simply is not doing what you want with it.

Written as a rule: retrieval changes what the model knows, fine-tuning changes how it behaves.

Retrieval wins on the practical dimensions that matter after launch. Content updates are a write to an index rather than a training run, so a policy change takes effect immediately. Answers can carry citations, which matters enormously for support, legal, and regulated contexts where an unsourced answer is unusable.

It also fails more gracefully. When retrieval finds nothing relevant, a well-built system says so and escalates. A fine-tuned model with a knowledge gap tends to answer confidently and wrongly, which is the more expensive failure.

Access control is the argument that settles it in most enterprise work. Retrieval can filter by the requesting user's permissions at query time. Knowledge baked into weights cannot be unlearned per user — once a model has trained on a document, you cannot show it to some users and not others.

There are real cases, and they cluster around consistency and cost rather than knowledge.

Rigid output structure at high volume is the strongest one. If you classify millions of documents into a fixed taxonomy, a fine-tuned smaller model can match a much larger prompted model at a fraction of the per-request cost and latency, because the instructions live in the weights instead of being resent with every call.

Voice and style are the second. If output must consistently match a specific editorial register, examples in a prompt drift and a few hundred good training examples do not.

Specialised notation is the third — legal citation conventions, clinical shorthand, industrial coding schemes — where the base model's grasp is shaky and no amount of retrieved context teaches the convention reliably.

In every one of these, we still put retrieval underneath for the facts. Fine-tuning shapes the response; retrieval supplies the truth.

A production retrieval system is mostly unglamorous engineering: ingestion and chunking, an embedding pipeline, a vector store, retrieval and reranking, and evaluation. Expect four to eight weeks for a first production version on a well-defined corpus, and remember that document preparation usually consumes more time than the model work.

Fine-tuning shifts the cost to data. The training run is often trivial — tens or low hundreds of euros for a small model. Producing several hundred to a few thousand clean, consistent, genuinely representative examples is the real expense, and it recurs whenever the task definition changes.

At inference, a fine-tuned small model is usually cheaper per request because prompts are shorter. Retrieval adds embedding and search cost plus more input tokens. At low volume that difference is noise; at millions of requests a month it becomes the dominant line item and is worth modelling properly before choosing.

Before either, exhaust prompting. Clear instructions, a handful of well-chosen examples, and structured output constraints solve a surprising share of problems that teams arrive convinced require fine-tuning.

Modern models follow instructions well enough that few-shot prompting often reaches the quality bar on its own. The sequence we recommend is prompting, then retrieval if facts are missing, then fine-tuning only if a measured behaviour gap remains.

Skipping straight to fine-tuning is the most common expensive mistake in this area. It is the option with the highest setup cost and the least flexibility, and it is frequently chosen for a problem that better instructions would have fixed in an afternoon.

Build a set of a few hundred real inputs with known good outputs before you change the architecture. Without it, every comparison is anecdote and every change is a guess.

Score the candidate approaches on the same set. In our work the result is frequently that retrieval alone clears the bar, which saves the fine-tuning budget entirely. When it does not, the evaluation tells you specifically where behaviour falls short, which is exactly the signal you need to build good training data.

Keep the set and run it on every prompt, model, and index change. This is the single practice that separates AI features that hold up over a year from ones that quietly degrade as models are swapped and content shifts.