A VP of Support asked me, in a room where three engineers nodded along, can we just train the model on our documentation? It is a reasonable question and it contains a hidden assumption that costs teams months: that putting knowledge into a model and putting knowledge in front of a model are the same operation. They are not, and picking the wrong one is the most expensive mistake available to you at this stage of a project.
Key takeaways
Retrieval is not the cheap option per request. Adding five 600 token chunks to our documentation assistant took cost from $5.22 to $11.22 per thousand questions on Claude Sonnet 5, a 2.1x increase.
What retrieval buys is a cheap unit of change. A documentation update costs minutes of re-embedding. A fine tuning refresh costs days of dataset curation, training and re-evaluation.
Under our modelled assumptions the two paths cost about the same money at roughly 65,000 questions per month. Below that, retrieval wins on money as well as freshness. Above it, the arithmetic flips and you should re-run it with your own numbers.
Availability is now part of the decision. OpenAI closed self-serve fine tuning to new organisations on 7 May 2026, and active customers lose the ability to create new jobs on 6 January 2027.
Two different questions, often confused
Most teams inherit a three rung ladder: start with prompting, add retrieval if you need fresh knowledge, fine tune when you outgrow both. That ladder is drawn on the wrong axis. It implies fine tuning is the advanced move you graduate to, and that retrieval is a stopgap on the way. In practice these techniques answer different questions, and you can need both, either or neither.
Fine tuning adjusts weights. It is a good instrument for changing how a model responds: the register it writes in, the shape of its output, a classification boundary it keeps getting wrong, a refusal pattern that is too eager. Retrieval changes what the model can see at the moment it answers. If your problem is that the assistant does not know the current refund window, that is a visibility problem, not a weights problem. Our GenAI Series comparison of fine tuning, RAG and prompting covers the conceptual split; here I am interested in which one you should spend money on, and why the answer is retrieval far more often than the ladder suggests.
| Property | Retrieval | Fine tuning |
|---|---|---|
| What it changes | What the model can see | How the model behaves |
| Unit of change | One document, re-embedded in seconds | One training run, hours to days |
| Cost profile | Higher per request, forever | Large up front, then flat |
| Attribution | Citable; you can show the source passage | None; the answer comes from weights |
| Removing a fact | Delete the document, effect is immediate | Retrain, and hope |
| Access control | Enforceable per user at query time | Baked in for everyone |
| Portability | Corpus survives a provider switch | Tied to one base model and vendor |
Read down the Attribution and Access control rows and you will see why regulated industries reach for retrieval almost by reflex. A support assistant that cannot show which paragraph it answered from is unauditable, and a fine tuned model that has memorised an enterprise customer contract will happily recite it to a different customer. Neither problem has a good fix once you are on the weights path.
Evidence on knowledge injection
Ovadia and colleagues ran the comparison directly, evaluating unsupervised fine tuning against retrieval on knowledge intensive tasks across several topics. Their finding is blunt: retrieval consistently outperformed fine tuning, both for knowledge the model had already seen in pre-training and for knowledge that was entirely new to it. Fine tuning did improve on the base model. It simply never caught up. They also report that models struggle to absorb new factual information through unsupervised fine tuning at all, and that showing the same fact in many paraphrased forms during training helps, which is a polite way of saying you need far more data than you expect to teach a model one fact reliably.
Sitting behind that result is catastrophic forgetting, which the same paper flags: additional training after pre-training can cost a model capabilities it previously had. You set out to teach it your refund policy and you quietly degrade something you were not measuring. If that sounds like the model decay discussed in Part 25 of the Data Science Series, it is the same class of problem, with a worse diagnostic story, because you have no feature distributions to plot.
I learned this the way most people do. Early in a support automation project I spent 11 days and roughly $4,000 of GPU time fine tuning a model on 6,200 curated ticket exchanges, because the stakeholder wanted the assistant to sound like our best support engineer. It worked. Tone was genuinely better and reviewers preferred it. It also began confidently quoting a pricing tier we had retired 14 months earlier, because that tier appeared about 400 times across the training tickets and three times in the current documentation. Weight of evidence in the training set beat truth in the docs, every time. We reverted to a strong prompt plus retrieval over an afternoon, and the retired tier stopped appearing entirely, because the retired tier was not in the corpus we were retrieving from. Eleven days bought us a tone improvement we later got from about forty lines of prompt.
Cost of change for the documentation assistant
Last part the assistant became reliable: it retries sensibly, respects rate limits and fails loudly rather than silently. What it still cannot do is answer a question about our product, because it has never seen our product. This part decides how that knowledge arrives, and from Part 8 onward we build whatever this section concludes.
Here is the claim I want to contradict, because I have heard it in three separate architecture reviews: retrieval is the cheap option. It is not. Retrieval makes every single request more expensive, permanently. Our prompt-only baseline runs an 800 token system prompt plus a 60 token question and returns about 350 tokens. On Claude Sonnet 5 at $2 per million input tokens and $10 per million output, that is $5.22 per thousand questions. Add five retrieved chunks of 600 tokens each and input jumps to 3,860 tokens, taking the same thousand questions to $11.22. Retrieval did not save money. It multiplied the per request bill by 2.1, and it will keep doing that for the life of the product.
| Assumption | Retrieval path | Fine tuning path |
|---|---|---|
| Extra cost per 1,000 questions | $6.00 | $0.00 |
| Volume assumed | 40,000 questions per month | 40,000 questions per month |
| Annual inference premium | $2,880 | $0 |
| Keeping content current | 2 doc updates per month, 0.25 hours each | Quarterly retrain, 14 hours each |
| Engineer cost assumed | $80 per hour loaded | $80 per hour loaded, plus $180 compute per run |
| Annual change cost | $480 | $5,200 |
| Twelve month total | $3,360 | $5,200 |
| Worst case content staleness | Under an hour | Up to three months |
Where the arithmetic flips
Solve for the volume at which the retrieval premium equals the fine tuning path and you get roughly 65,000 questions per month. Below that, retrieval wins on money and on freshness. Above it, retrieval is still probably right, but you are now paying for the privilege and you owe your finance team a reason. Prompt caching is the lever that moves this line: at a 0.1x cache read multiplier, a stable retrieved prefix substantially reduces the premium, which is why Part 26 revisits this calculation rather than treating it as settled.
Where fine tuning still wins
A survey that only argues one side is not worth reading, so here are the cases where I have seen fine tuning earn its cost. First, high volume narrow classification. If you are routing two million tickets a month into eleven categories, a small fine tuned model can replace a large prompted one and cut both latency and spend by an order of magnitude. Second, output format under zero tolerance. When a downstream system will break on malformed output and prompt engineering has plateaued around 99.4 percent validity, training closes that last stretch in a way instructions do not. Third, a house style that resists description. Some tone requirements are easier to demonstrate with 2,000 examples than to specify in prose, and the war story above is exactly this case: the tone half of that project was legitimate, the knowledge half was not.
Notice what those three share. None of them is about facts. Each is about behaviour, at a volume or a tolerance where behaviour is worth buying outright. Notice also that all three become clear only after you have measured something, which is why the honest sequence is prompt, measure, retrieve, measure again, and only then consider weights.
Availability risk you cannot engineer around
A new argument arrived in 2026 that did not exist when most fine tuning guidance was written. OpenAI has been winding down its self-serve fine tuning platform. On 7 May 2026, organisations that had not previously fine tuned lost the ability to create training jobs at all. On 2 July 2026, organisations that had not run inference on a fine tuned model in the preceding 60 days lost it too. Active existing customers lose the ability to create new fine tuning jobs on 6 January 2027, with inference on already fine tuned models continuing only until the underlying base model is deprecated. Their stated reasoning is that newer base models follow instructions and formats well enough that much of the fine tuning process is no longer necessary.
This series is deliberately vendor neutral, and the picture differs elsewhere: fine tuning remains available on the open weight ecosystem, on Bedrock, on Vertex AI and on Azure, so nobody is locked out. What has changed is the risk profile. A fine tuned model was already tied to one base model snapshot; that snapshot will be deprecated, and your customisation dies with it. Now the platform itself may be withdrawn on a published timetable. A retrieval corpus has neither problem. Move providers and you re-embed, which costs money and a weekend. Move providers with a fine tuned model and you start the whole exercise again. Part 29 treats provider portability properly; for now it is enough to note that the portable choice and the cheap choice happen to be the same choice.
Symptom to approach lookup
Stakeholders rarely report a technique gap. They report a symptom, and the translation from symptom to technique is where most of the judgement lives. Keep this table where you triage feedback.
| What the complaint sounds like | What it actually is | Reach for |
|---|---|---|
| It does not know about our new plan tiers | Missing, changing fact | Retrieval |
| It invents config options that do not exist | No grounding, and no refusal instruction | Retrieval plus an abstain rule in the prompt |
| It sounds nothing like us | Behaviour gap | Prompt first; fine tuning if prompting plateaus |
| Output breaks our parser once in every 200 calls | Format reliability at the tail | Schema validation, then fine tuning at volume |
| Answers are right but we cannot prove where they came from | Attribution requirement | Retrieval, non negotiable |
| Contractors can see answers drawn from internal runbooks | Access control | Retrieval with per user filtering at query time |
| Every call costs too much at our volume | Per request economics | Caching and a smaller model, then fine tuning |
| Answers were correct last quarter and are wrong now | Staleness, from any baked in knowledge | Retrieval, and delete whatever baked it in |
One caution on that table. Retrieval is not a correctness guarantee, only a grounding mechanism; a model handed the right passage can still summarise it wrongly, and the temperature and sampling behaviour covered in the GenAI Series piece on hallucination does not disappear because you added context. Part 13 separates retrieval quality from answer quality precisely because teams conflate them and then blame the wrong component.
Build retrieval, and price it honestly
My recommendation for this project, and for most business assistants that answer from documents, is to build retrieval and to stop treating fine tuning as an inevitable later phase. Retrieval will make each request cost roughly twice what it did. Say that out loud in the design review, put the number in the document, and pair it with the number that justifies it: a documentation change reaches users in under an hour instead of at the mercy of the next training run.
On Monday, do one thing. Take your last twenty pieces of user feedback about your assistant, run each one through the symptom lookup above, and count how many land on retrieval, how many land on prompting and how many genuinely land on fine tuning. I have run this exercise with four teams and the fine tuning column has been empty three times. If yours is empty too, you have just saved a quarter, and Part 8 starts the real work: getting your documents into a state a retrieval system can actually use, which is messier than anyone admits.
References
- Ovadia, Brief, Mishaeli and Elisha, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, arXiv:2312.05934v3, for the finding that retrieval consistently outperforms unsupervised fine tuning on both previously seen and entirely new knowledge, that models struggle to learn new facts through unsupervised fine tuning, that paraphrase variety during training mitigates this, and for the catastrophic forgetting discussion.
- OpenAI API deprecations, for the self-serve fine tuning wind down dates of 7 May 2026, 2 July 2026 and 6 January 2027, and the statement that inference on fine tuned models continues until the base model is deprecated.
- Claude Platform docs, Pricing, for the Claude Sonnet 5 rates of $2 and $10 per million input and output tokens in effect through 31 August 2026, and the 0.1x cache read multiplier used in the caching note.
- kubernetes/website, the public Markdown corpus standing in for product documentation across this series, and the corpus Part 8 begins ingesting.
About The Author
DrJha