mem0 vs MemPalace: The Benchmark Numbers Are Not Measuring the Same Thing

Two README files, both read on 2 September 2026. mem0 headlines 94.4 on LongMemEval. MemPalace headlines 96.6% on LongMemEval. Same benchmark name, a 2.2 point gap, and an obvious conclusion waiting to be drawn.

The conclusion is wrong, and both projects say so in their own documentation.

mem0’s 94.4 is end-to-end question answering accuracy, measured on the managed platform.

Its README adds this line directly under the table: “Scores reflect Mem0’s managed platform, which includes proprietary optimizations not available in the open-source SDK; open-source users should expect directionally similar gains but not identical numbers.”

MemPalace’s 96.6% is retrieval recall at k=5. It measures whether the right session appeared in the top five results. No model reads them. No answer is generated.

And the MemPalace README refuses to put the two side by side: “We deliberately do not include a side-by-side comparison against Mem0, Mastra, Hindsight, Supermemory, or Zep.

Those projects publish different metrics on different splits, and placing retrieval recall next to end-to-end QA accuracy is not an honest comparison.”

So the question everybody types into Google, which one scores higher, cannot be answered from published numbers. Nobody has run both systems on the same split with the same metric and the same reader model.

Until somebody does, every ranking table you find that puts these two in one column is comparing a distance to a temperature.

What you can compare is architecture. And architecture decides the two things that actually bite you later: what it costs to save something, and whether you can change your mind in a year.

The one difference everything else follows from

mem0 calls a language model when you save. MemPalace does not.

That is the whole comparison. Everything below is a consequence of it.

mem0’s write path

mem0’s write path, as of the April 2026 algorithm described in its README, is single-pass ADD-only extraction: one LLM call per add, no update, no delete.

The model reads your conversation, pulls out what it considers facts, and those facts get embedded and stored. Entities are extracted and linked across memories.

Retrieval then fuses three signals in parallel: semantic similarity, BM25 keyword matching, and entity matching. The documented default model is gpt-5-mini and the documented default embedder is OpenAI’s text-embedding-3-small.

MemPalace’s write path

MemPalace’s write path stores your content verbatim. Its README is blunt about it: “It does not summarize, extract, or paraphrase.” The index is structured rather than flat.

People and projects become wings, topics become rooms, and the original text lives in drawers, so a search can be scoped to a wing instead of run against the whole corpus.

Embeddings are computed locally by a model that ships to your disk: roughly 30 MB for all-MiniLM-L6-v2, roughly 300 MB for embeddinggemma-300m. The README states plainly that no API key is required for the core benchmark path.

What extraction costs you that is not money

The money part is easy and I work it out with published rates in the post on what self-hosted AI memory actually costs to run. The part people miss is that extraction is lossy in a way that is not recoverable.

The model becomes part of your data

When an LLM decides what counts as a fact in your Slack thread, the memory you keep is a function of that model’s judgement on that day.

Swap the model in eighteen months and the same input produces a different set of memories. You cannot re-derive the old ones, because the source text was never the thing you stored.

Verbatim storage inverts this. The extraction, if you want one, happens at read time against whatever model you have then. You can re-index. You can change embedding models.

MemPalace documents exactly that operation: switching to a server-side embedding endpoint requires mempalace repair rebuild-index, because a different embedder means a different vector space.

ADD-only means nothing is overwritten

There is a second consequence hiding in mem0’s design note. ADD-only means “memories accumulate; nothing is overwritten”. If you told it in March that you use Postgres and in August that you moved to SQLite, both facts are in the store.

Retrieval has to rank them, and ranking is where temporal reasoning either works or does not. mem0’s April 2026 notes describe time-aware retrieval that ranks the right dated instance.

MemPalace handles the same problem in a separate place: a temporal entity-relationship graph with validity windows, backed by local SQLite, with add, query, invalidate and timeline operations.

Side by side, on facts you can check

Note the star counts. When I published the full comparison of AI memory tools for a second brain, mem0 was at 53k and MemPalace at 46.7k.

Both have moved by more than ten thousand stars since. If a comparison page you are reading quotes the old numbers, it has not been rechecked, which tells you something about the rest of it.

Check the domain before you trust the benchmark

The MemPalace README carries a warning I have not seen on any other project in this category:

Beware of impostor sites. MemPalace has no other official websites. The only official sources are this GitHub repository and the docs at mempalaceofficial.com. Any other domain (including .tech, .net, or other .com variants) is an impostor and may distribute malware.

MemPalace README, read 2 September 2026

Why that warning matters here

That matters here because several of the ranking pages that surface for “best AI memory framework” are hosted on exactly those lookalike domains, and their benchmark tables are where a lot of the numbers in circulation came from.

If you want MemPalace’s figures, take them from benchmarks/BENCHMARKS.md in the repository, which commits the per-question result files and the commands to reproduce them.

What the benchmark file actually says

Read that file and you get a more useful picture than the headline. Raw semantic search with no heuristics and no LLM scores 96.6% R@5.

A tuned hybrid pipeline, evaluated on a held-out 450 questions after tuning on 50, scores 98.4%. Adding an LLM reranker over the top 20 gets to 99% or better.

The README declines to headline the 100% figure because reaching it involved inspecting specific wrong answers, which it flags as teaching to the test.

A project that argues itself down from 100% is a project whose 96.6% I am inclined to believe.

The things that will actually break

Both projects document their own sharp edges. These are the ones that cost an evening.

• mem0 self-hosted auth is on by default. The README flags it: upgrading from a pre-auth build needs ADMIN_API_KEY set, an admin registered through the wizard, or AUTH_DISABLED=true for local development only.
• mem0 hybrid search is an extra install. Plain pip install mem0ai gives you semantic retrieval. BM25 keyword matching and entity extraction need pip install mem0ai[nlp] plus python -m spacy download en_core_web_sm. It is easy to benchmark the wrong configuration and conclude the retrieval is weak.
• MemPalace on Android does not work natively. The README states that Termux is unsupported because ChromaDB and ONNX Runtime publish Linux wheels, not Android wheels. The tested path is a Debian PRoot container.
• The MemPalace GPU image is x86_64 only. onnxruntime-gpu publishes no aarch64 Linux wheels, so docker build -f Dockerfile.gpu fails on Apple Silicon with a dependency resolution error rather than anything that names the real cause.
• MemPalace Docker on Linux hits a permissions wall. The image runs as uid 1000 and bind mounts keep host ownership, so a 0700 directory surfaces as PermissionError: [Errno 13]. The README specifically warns against working around it with --user, because /data inside the image is owned by uid 1000 and no other uid can write the palace.
• First run needs network on both. MemPalace downloads its embedding model into /data on the first call that needs embeddings. A slow first container is not a hung one.

Running either one against a coding agent

This is where the two projects diverge most in practice, and it is the use case most people in this search actually have.

MemPalace with a coding agent

MemPalace ships auto-save hooks for Claude Code, Codex CLI and Cursor IDE that save periodically and before context compression, plus 45 MCP tools covering palace reads and writes, knowledge graph operations, cross-wing navigation, drawer management and agent diaries.

Its README carries a standing warning that Claude Code sessions expire in 30 days without those hooks wired, and points at a retention checklist for backfilling existing JSONL transcripts with mempalace mine ~/.claude/projects/ --mode convos.

# MemPalace, install and first index
uv tool install mempalace
mempalace init ~/projects/myapp
mempalace mine ~/.claude/projects/ --mode convos
mempalace search "why did we switch to GraphQL"
mempalace wake-up

mem0 with a coding agent

mem0 approaches the same problem through skills and a CLI.

Its README documents reference skills that load SDK knowledge into an assistant’s context and pipeline skills that run an end-to-end workflow in an existing repo, plus an agent signup flow that mints a working API key without an email or a dashboard.

# mem0, agent-first signup as documented in the README
npm install -g @mem0/cli
mem0 init --agent --agent-caller claude-code
mem0 add "I am using mem0"
mem0 search "am I using mem0"

Both plug into an MCP client.

If you are wiring either of these into a working agent setup, the practical constraints on tool count and permissions are the same ones I hit in running Claude Code against production WordPress.

Forty-five tools is not free: every tool’s name, description and input schema sits in context on every request.

What I cannot tell you

I have not benchmarked these two against each other, and I am not going to publish a table that implies I did.

A fair head-to-head needs the same LongMemEval split, the same metric, the same reader model, and the same hardware, run by somebody with no stake in the result.

That run does not exist publicly as of 2 September 2026.

Retrieval latency

I also cannot tell you retrieval latency for either. Both projects publish figures under their own conditions. Neither published a methodology detailed enough to reproduce the other’s. Anyone quoting a millisecond comparison between them is quoting two different experiments.

What is checkable is everything in the table above, and the checkable part is enough to decide.

It is the same discipline I applied when a commercial security scanner reported a site clean while it was serving spam to Googlebot: the vendor’s headline number and the vendor’s mechanism are two different claims, and only one of them can be verified from the outside.

Pick by answering one question

Does a language model need to see every item at the moment you save it?

If yes, because you want structured facts extracted for a chatbot that will be queried by thousands of end users, mem0 is built for that shape and has the larger ecosystem to prove it.

If no, because you are storing your own working history and you want the source text to survive the next three model generations, MemPalace’s write path is the one that does not charge you per save.

Then go and check the arithmetic against your own volume before you commit. Two numbers decide it: items saved per month, and average tokens per item.

More on ai memory and mcp

• Cognee Alternatives: What You Are Actually Replacing
• What Self-Hosted AI Memory Actually Costs to Run
• MCP Server vs RAG: Which One You Actually Need

Resources

• mem0 repository and README: github.com/mem0ai/mem0
• mem0 documentation: docs.mem0.ai
• mem0 paper, Chhikara et al. 2025: arXiv 2504.19413
• MemPalace repository and README: github.com/MemPalace/mempalace
• MemPalace official docs: mempalaceofficial.com
• LongMemEval, the benchmark both cite: arXiv 2410.10813