Aditya Sharma

RAG vs Fine-Tuning: Which Problem Do You Have?

On this page, 5 sections

If your agent gets facts wrong about your own world, fix the context with retrieval. If it knows the facts and still answers in the wrong shape, tone or format, that is the case for fine-tuning. Most agent problems I see are the first kind.

I should be plain about my side of this. I have built retrieval and memory layers for agents. I have not fine-tuned a model. So this post is how I would judge the choice, built on OpenAI’s own optimization guide, which I read on 17 November 2026.

OpenAI's accuracy guide. A matrix puts context optimization (what the model needs to know) on the path to RAG and LLM optimization (how the model needs to act) on the path to fine-tuning. The text below says the context axis maximizes response accuracy and the LLM axis maximizes consistency of behavior. Source: OpenAI, Optimizing LLM accuracy. Headline: Missing facts or wrong behaviour?

The two problems people mix up

The OpenAI guide frames it as two axes, not a ladder. Context optimization is for when the model lacks knowledge because it was not in its training set, its knowledge is out of date, or it needs proprietary information. LLM optimization is for when results are inconsistent, the tone or format is wrong, or the reasoning is not followed consistently.

The guide says directly that many how-to posts draw a straight line from prompting to retrieval to fine-tuning, and that this is often not the case. They are levers that solve different things. That matches how I think about it: first work out which problem you have, then pick the lever.

SymptomLeverWhy
Agent does not know my notes, docs or setupRetrieval (RAG)The missing thing is knowledge
Agent knows the answer but ignores the formatFine-tuning, after prompt and examplesThe missing thing is behaviour
Facts change every weekRetrievalRetrieval reads today’s data, no retraining
Same tone needed across thousands of outputsFine-tuningConsistency is the goal
My reading of the two axes in OpenAI’s Optimizing LLM Accuracy guide, opened 17 November 2026. The last two rows are my judgement, not a quote.

Why I start with the context

My view: an agent working on my own sites and notes rarely fails because it cannot write. It fails because it does not know what is in my folder, my last decision or my current numbers. That is a knowledge problem, so retrieval and memory are where I spent my time.

If you want the memory side, I compared the main options in AI memory tools compared and Graphiti vs Cognee.

The guide also warns about the failure that bites retrieval: you can supply the wrong context so the model cannot answer, or too much irrelevant context that drowns out the real information and causes hallucinations. I read that as a reason to keep what you retrieve small and checked, which is also how I try to keep an agent’s memory.

When I would reach for fine-tuning

Only after prompting and examples have failed, and only for a behaviour problem. The guide says to start with prompt engineering and an evaluation set, and its worked example prepares 50+ examples before a fine-tune.

It also says to start small and focus on quality, and to make training examples match what the model sees in production. If your app uses RAG, it says to fine-tune with RAG examples in it.

That last point is the one I would not want to learn the hard way. A fine-tune built on tidy examples, used behind a messy retrieval step, teaches the model something your real traffic never looks like.

The guide also has a line I agree with: squeeze as much accuracy as you can out of the basic methods before reaching for RAG plus fine-tuning, and let your accuracy target be the objective, not the sophistication.

What I could not verify

  • How fine-tuning feels in practice. I have not done it, so I will not describe the process as if I had.
  • Any cost comparison. Prices and training limits change, and I did not check them today.
  • Whether the guide’s advice holds for every model. It is written around OpenAI’s offerings, and says the principles should hold for other providers and open source.

Before you build either, write down five questions your agent gets wrong. If the wrong answers are missing facts, start with retrieval. If the facts are right and the shape is wrong, you have a prompt and examples problem first. A fine-tune comes after that, not before.

Sources

Tell me where I am wrong

Your email is not published and I do not add it to any list. Corrections with a source are the ones I act on fastest.