MCP Server vs RAG: Which One You Actually Need

On this page, 8 sections

Google Ads reports 60,500 US searches a month for the phrase “mcp server”, at low competition (keyword data via DataForSEO, pulled 20 August 2026).

A large share of those people are trying to work out whether an MCP server replaces the RAG pipeline they were about to build.

It does not, because the two things are not the same kind of object. RAG is a retrieval architecture. MCP is a wire protocol.

You can serve RAG results through an MCP server, and plenty of people do. Asking which to pick is like asking whether to use a database or HTTP.

The real decision underneath the question is worth answering carefully, because getting it wrong is expensive in a way that shows up months later: should this data be pre-indexed and searched by meaning, or queried live by a tool while the model is running? That is a question about freshness and authority, and it has a clean answer per data source.

What each one actually is

RAG comes from Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, submitted 22 May 2020 and last revised 12 April 2021 (arXiv 2005.11401).

The pattern is retrieve from a non-parametric index, then condition generation on what you retrieved.

In current practice that means: chunk your corpus, embed each chunk, store the vectors, embed the query at request time, take the top k by similarity, paste those chunks into the prompt.

The load-bearing property of RAG

The load-bearing property is that retrieval happens before the model runs, and it is approximate. Similarity is not correctness.

What MCP is, at the wire

MCP is specified at modelcontextprotocol.io, protocol version 2026-07-28. It is JSON-RPC 2.0 over either stdio, for a local process, or Streamable HTTP, for a remote one.

It is stateless: every request carries the protocol version and the relevant capabilities in its _meta field, so a server can process each request on its own.

Servers advertise what they support through a mandatory server/discover request.

The spec defines three server primitives:

• Tools: executable functions the model can invoke, discovered with tools/list and run with tools/call.
• Resources: data sources that provide context, such as file contents, database records or API responses.
• Prompts: reusable templates, such as system prompts and few-shot examples.

The client primitive, and the two that are gone

And one client primitive that is still current, elicitation, which lets a server ask the user for input or confirmation via elicitation/create. Sampling and logging are both deprecated as of protocol version 2026-07-28.

The spec tells new implementations to integrate directly with LLM provider APIs instead of sampling, and to log to stderr or OpenTelemetry instead of the logging primitive.

The load-bearing property of MCP

The load-bearing property is that a tool call happens during the run, on the model’s decision, and it returns whatever the underlying system says right now.

The two shapes, side by side

Here is what the model sees in each case. First a tool, as returned by tools/list in the spec’s own example, with the field descriptions trimmed out.

{
  "name": "weather_current",
  "title": "Weather Information",
  "description": "Get current weather information for any location worldwide",
  "inputSchema": {
    "type": "object",
    "properties": {
      "location": { "type": "string" },
      "units": { "type": "string", "enum": ["metric", "imperial", "kelvin"], "default": "metric" }
    },
    "required": ["location"]
  }
}

The model reads a schema and decides whether to call it.

The answer it gets back is authoritative, because it came from the system that owns the fact.

And what RAG looks like from the model side

Now RAG. The model does not decide anything. Your code decides, before the model is invoked:

q_vec  = embed(user_question)
chunks = vectordb.search(q_vec, top_k=5)
prompt = system + "\n\nContext:\n" + "\n---\n".join(c.text for c in chunks) + "\n\n" + user_question

Five chunks that were similar to the question go in. If the right chunk ranked sixth, the model never sees it and will answer confidently from the five it got.

The decision rule

Go through your data sources one at a time and mark each of them against these five tests. This is the part of the exercise that pays.

If this is true of the source	Use	Because
It changes faster than you can re-index	Tool call	An index is a snapshot. Order status, stock levels, today’s logs, a live analytics query
A wrong answer is worse than no answer	Tool call	Similarity search on a price list returns a price similar to the right one
Large, static, and questions are fuzzy	RAG	Documentation, past tickets, six years of your own writing
Users ask about things nobody indexed by name	RAG	Semantic search finds the paragraph you never thought to tag
The action changes state	Tool call	Retrieval cannot refund an order. Only a tool can

Most working systems land on both, and the split falls out of the table rather than out of a preference. A support agent does RAG over the knowledge base and a tool call for the customer’s actual account.

A coding agent does RAG over the codebase and a tool call for the current test output.

The row people underrate

The second row of that table is the one people underrate. A pre-built index is a claim about the past. When the claim and the live system disagree, the index wins by default and nobody finds out.

I watched that exact failure play out with commercial malware scanners: every scanner reported the site clean while it was serving spam to Googlebot, because they were matching against a signature index instead of asking the live server what it returns to a Googlebot user agent. Same architecture mistake, different domain.

The split I run

That is the shape of the stack I ended up with in running Claude Code against production WordPress: live tool calls for anything the site knows about itself, and indexed retrieval for the documentation and history that does not change hour to hour.

Where “just wrap it in MCP” goes wrong

These are the failure modes worth knowing before you build, all of them grounded in what the spec actually says.

Wrapping a vector store in an MCP server and calling it something else

If your MCP tool is search_docs(query) and it does an embedding lookup, you built RAG with an extra network hop. That is a legitimate thing to build.

It is not an alternative to RAG, and the retrieval quality problems do not go away because JSON-RPC is now involved. MCP moves bytes.

It has no opinion about whether your chunks are any good.

Tool descriptions are not free

Every tool’s name, title, description and full input schema is text the model reads. MemPalace, one of the AI memory servers I looked at in the comparison of AI memory tools for a second brain, ships 45 MCP tools.

Graphiti’s server exposes episode management, entity management, search and graph maintenance. Connect three servers like that and a meaningful slice of your context window is gone before the user has typed anything.

The spec gives you tools/list pagination through an optional cursor parameter for a reason.

Handing the model raw SQL against a schema it has not seen

The spec’s own worked example is the fix: a database server exposes tools for querying, a resource that contains the schema, and a prompt with few-shot examples for using the tools.

Three primitives, one job. Most homemade database MCP servers ship only the first and then blame the model.

Building on primitives that are on the way out

Sampling, which let a server ask the client’s application for a model completion so the server did not need its own LLM SDK, is deprecated as of 2026-07-28.

If your design assumed the client would provide inference, that assumption expired. A lot of tutorials written earlier still teach it.

Assuming the client notices when your tool list changes

Change notifications are opt-in. Per the spec, the client opens a long-lived subscriptions/listen stream naming the notification types it wants, and the server delivers matching notifications on that stream.

Declaring "tools": {"listChanged": true} in your capabilities means you can honour that filter. It does not mean anyone is listening.

Assuming a local server scales like a remote one

Local servers on stdio typically serve a single client. Remote servers on Streamable HTTP typically serve many, and carry standard HTTP authentication, with the spec recommending OAuth for tokens.

A stdio server that works beautifully on your laptop is not a deployment.

The comparison I am not going to fake

You will find posts with latency tables putting “MCP tool call” against “RAG retrieval” in milliseconds. Ignore them unless they publish the corpus, the model, the hardware and the network path, because the numbers are otherwise meaningless.

A tool call to a local SQLite file and a tool call to a rate-limited third-party API are the same row in that table and differ by three orders of magnitude.

The cost shape, which is the part that decides your bill

I have not run that benchmark and I am not going to publish one I did not run. What I can give you is the cost shape, which is the part that actually determines your bill:

• RAG cost is paid at index time and is roughly fixed per document. Embedding a million tokens with OpenAI’s text-embedding-3-small costs $0.02 at the rates published on developers.openai.com, read 2 September 2026. You pay again only when you re-index.
• Tool call cost is paid per query and scales with traffic. It is usually not the API call that costs, it is the extra model turns: the model calls the tool, reads the result, then answers, so you pay for the context twice.

I work both of those through with published per-token prices, at a stated volume, in what self-hosted AI memory actually costs to run.

One thing to do next

List every data source your agent needs. Next to each one write a single word: hourly, weekly, or yearly, meaning how often the truth in it changes.

What to do with the answers

Everything marked hourly is a tool. Everything marked yearly is an index.

The weekly ones are where the interesting engineering is, and they are usually best served as an index with a tool that can check a single record on demand when the answer matters.

More on ai memory and mcp

• Cognee Alternatives: What You Are Actually Replacing
• mem0 vs MemPalace: The Benchmark Numbers Are Not Measuring the Same Thing
• What Self-Hosted AI Memory Actually Costs to Run

Resources

• MCP specification, current version: modelcontextprotocol.io/specification/latest
• MCP architecture overview, primitives and the worked JSON-RPC example: modelcontextprotocol.io/docs/learn/architecture
• MCP Inspector, for testing a server before you wire it to a client: github.com/modelcontextprotocol/inspector
• Reference server implementations: github.com/modelcontextprotocol/servers
• The original RAG paper, Lewis et al.: arXiv 2005.11401
• OpenAI model and embedding prices: platform.openai.com/docs/pricing