RAG vs Fine-Tuning: Which Problem Do You Have?
On this page, 5 sections
If your agent gets facts wrong about your own world, fix the context with retrieval. If it knows the facts and still answers in the wrong shape, tone or format, that is the case for fine-tuning. Most agent problems I see are the first kind.
I should be plain about my side of this. I have built retrieval and memory layers for agents. I have not fine-tuned a model. So this post is how I would judge the choice, built on OpenAI’s own optimization guide, which I read on 17 November 2026.

The two problems people mix up
The OpenAI guide frames it as two axes, not a ladder. Context optimization is for when the model lacks knowledge because it was not in its training set, its knowledge is out of date, or it needs proprietary information. LLM optimization is for when results are inconsistent, the tone or format is wrong, or the reasoning is not followed consistently.
The guide says directly that many how-to posts draw a straight line from prompting to retrieval to fine-tuning, and that this is often not the case. They are levers that solve different things. That matches how I think about it: first work out which problem you have, then pick the lever.
| Symptom | Lever | Why |
|---|---|---|
| Agent does not know my notes, docs or setup | Retrieval (RAG) | The missing thing is knowledge |
| Agent knows the answer but ignores the format | Fine-tuning, after prompt and examples | The missing thing is behaviour |
| Facts change every week | Retrieval | Retrieval reads today’s data, no retraining |
| Same tone needed across thousands of outputs | Fine-tuning | Consistency is the goal |
Why I start with the context
My view: an agent working on my own sites and notes rarely fails because it cannot write. It fails because it does not know what is in my folder, my last decision or my current numbers. That is a knowledge problem, so retrieval and memory are where I spent my time.
If you want the memory side, I compared the main options in AI memory tools compared and Graphiti vs Cognee.
The guide also warns about the failure that bites retrieval: you can supply the wrong context so the model cannot answer, or too much irrelevant context that drowns out the real information and causes hallucinations. I read that as a reason to keep what you retrieve small and checked, which is also how I try to keep an agent’s memory.
When I would reach for fine-tuning
Only after prompting and examples have failed, and only for a behaviour problem. The guide says to start with prompt engineering and an evaluation set, and its worked example prepares 50+ examples before a fine-tune.
It also says to start small and focus on quality, and to make training examples match what the model sees in production. If your app uses RAG, it says to fine-tune with RAG examples in it.
That last point is the one I would not want to learn the hard way. A fine-tune built on tidy examples, used behind a messy retrieval step, teaches the model something your real traffic never looks like.
The guide also has a line I agree with: squeeze as much accuracy as you can out of the basic methods before reaching for RAG plus fine-tuning, and let your accuracy target be the objective, not the sophistication.
What I could not verify
- How fine-tuning feels in practice. I have not done it, so I will not describe the process as if I had.
- Any cost comparison. Prices and training limits change, and I did not check them today.
- Whether the guide’s advice holds for every model. It is written around OpenAI’s offerings, and says the principles should hold for other providers and open source.
Before you build either, write down five questions your agent gets wrong. If the wrong answers are missing facts, start with retrieval. If the facts are right and the shape is wrong, you have a prompt and examples problem first. A fine-tune comes after that, not before.