--- title: "mem0 vs MemPalace: The Benchmark Numbers Are Not Measuring the Same Thing" url: https://adityaarsharma.com/mem0-vs-mempalace/ date: 2026-09-16 modified: 2026-09-03 author: "Aditya Sharma" description: "94.4 and 96.6 on the same benchmark name, measuring two different quantities. What is actually comparable, and what decides it." categories: - "AI" - "Automation" image: https://adityaarsharma.com/wp-content/uploads/2026/09/755362f2-878e-4fb0-bf04-2d2984c8eafb_2912x1632-1024x574.webp word_count: 1984 --- # mem0 vs MemPalace: The Benchmark Numbers Are Not Measuring the Same Thing Two README files, both read on 2 September 2026. mem0 headlines **94.4 on LongMemEval**. MemPalace headlines **96.6% on LongMemEval**. Same benchmark name, a 2.2 point gap, and an obvious conclusion waiting to be drawn. ![One of these two systems calls a language model every time you save something.](https://adityaarsharma.com/wp-content/uploads/2026/09/755362f2-878e-4fb0-bf04-2d2984c8eafb_2912x1632-scaled.png)The conclusion is wrong, and both projects say so in their own documentation. ![mem0 README benchmark table showing LoCoMo, LongMemEval and BEAM scores with a note that the numbers reflect the managed platform](https://adityaarsharma.com/wp-content/uploads/2026/09/67e4e765-deee-4279-aa71-b153adce6e2d_2800x1720-scaled.png)The mem0 README benchmark table on GitHub, read 3 September 2026. The caveat under it says the scores reflect the managed platform, which includes proprietary optimisations.mem0's 94.4 is end-to-end question answering accuracy, measured on the managed platform. Its README adds this line directly under the table: "Scores reflect Mem0's managed platform, which includes proprietary optimizations not available in the open-source SDK; open-source users should expect directionally similar gains but not identical numbers." MemPalace's 96.6% is retrieval recall at k=5. It measures whether the right session appeared in the top five results. No model reads them. No answer is generated. And the MemPalace README refuses to put the two side by side: "We deliberately do not include a side-by-side comparison against Mem0, Mastra, Hindsight, Supermemory, or Zep. Those projects publish different metrics on different splits, and placing retrieval recall next to end-to-end QA accuracy is not an honest comparison." So the question everybody types into Google, which one scores higher, cannot be answered from published numbers. Nobody has run both systems on the same split with the same metric and the same reader model. Until somebody does, every ranking table you find that puts these two in one column is comparing a distance to a temperature. What you can compare is architecture. And architecture decides the two things that actually bite you later: what it costs to save something, and whether you can change your mind in a year. On this page - यह नक्षत्र और पद असल में क्या दर्शाते हैं- यह प्लेसमेंट कैसे दिखता है- वह बात जो कम बताई जाती है- यह गुरु कब मजबूत या कमज़ोर होता है- लोग अक्सर गलत समझते हैं- महादशा में यह कब जागता है- यह पद कैसे निकाला जाता है- बुध का नक्षत्र-स्वामी होना क्या बदलता है- कुंडली में यह गुरु किस भाव में है, यह भी देखिए- अक्सर पूछे जाने वाले सवाल ## The one difference everything else follows from mem0 calls a language model when you save. MemPalace does not. That is the whole comparison. Everything below is a consequence of it. ### mem0's write path mem0's write path, as of the April 2026 algorithm described in its README, is single-pass ADD-only extraction: one LLM call per add, no update, no delete. The model reads your conversation, pulls out what it considers facts, and those facts get embedded and stored. Entities are extracted and linked across memories. Retrieval then fuses three signals in parallel: semantic similarity, BM25 keyword matching, and entity matching. The documented default model is `gpt-5-mini` and the documented default embedder is OpenAI's `text-embedding-3-small`. ### MemPalace's write path MemPalace's write path stores your content verbatim. Its README is blunt about it: "It does not summarize, extract, or paraphrase." The index is structured rather than flat. People and projects become *wings*, topics become *rooms*, and the original text lives in *drawers*, so a search can be scoped to a wing instead of run against the whole corpus. Embeddings are computed locally by a model that ships to your disk: roughly 30 MB for `all-MiniLM-L6-v2`, roughly 300 MB for `embeddinggemma-300m`. The README states plainly that no API key is required for the core benchmark path. ## What extraction costs you that is not money ![mem0 pricing page showing Hobby free, Starter 19 dollars, Pro 249 dollars and a custom Enterprise tier](https://adityaarsharma.com/wp-content/uploads/2026/09/f993c65e-41f4-4799-b8dc-37c9e4fbf4e8_2800x1720-scaled.png)mem0.ai pricing, read 3 September 2026. The managed platform is where the headline benchmark numbers were measured.The money part is easy and I work it out with published rates in [the post on what self-hosted AI memory actually costs to run](https://adityaarsharma.com/self-hosted-ai-memory-cost/). The part people miss is that extraction is lossy in a way that is not recoverable. ### The model becomes part of your data When an LLM decides what counts as a fact in your Slack thread, the memory you keep is a function of that model's judgement on that day. Swap the model in eighteen months and the same input produces a different set of memories. You cannot re-derive the old ones, because the source text was never the thing you stored. Verbatim storage inverts this. The extraction, if you want one, happens at read time against whatever model you have then. You can re-index. You can change embedding models. MemPalace documents exactly that operation: switching to a server-side embedding endpoint requires `mempalace repair rebuild-index`, because a different embedder means a different vector space. ### ADD-only means nothing is overwritten There is a second consequence hiding in mem0's design note. ADD-only means "memories accumulate; nothing is overwritten". If you told it in March that you use Postgres and in August that you moved to SQLite, both facts are in the store. Retrieval has to rank them, and ranking is where temporal reasoning either works or does not. mem0's April 2026 notes describe time-aware retrieval that ranks the right dated instance. MemPalace handles the same problem in a separate place: a temporal entity-relationship graph with validity windows, backed by local SQLite, with add, query, invalidate and timeline operations. ## Side by side, on facts you can check | Property | mem0 | MemPalace | | -------- | ---- | --------- | | Licence | Apache 2.0 | MIT | | GitHub stars, 2 Sep 2026 | 64.6k | 58.8k | | LLM call on save | Yes, one per add | No, default path | | Default embedder | OpenAI text-embedding-3-small | Local, MiniLM or embeddinggemma-300m | | What is stored | Extracted facts plus embeddings | Verbatim text, structured index | | Default store | Vector store, docker compose stack | ChromaDB, pluggable | | Other backends | See component docs | sqlite_exact, milvus, qdrant, pgvector | | Temporal handling | Time-aware retrieval ranking | SQLite graph with validity windows | | MCP surface | MCP server plus agent skills and CLI | 45 MCP tools | | Managed option | Yes, app.mem0.ai | None | Note the star counts. When I published [the full comparison of AI memory tools for a second brain](https://adityaarsharma.com/ai-memory-tools-compared/), mem0 was at 53k and MemPalace at 46.7k. Both have moved by more than ten thousand stars since. If a comparison page you are reading quotes the old numbers, it has not been rechecked, which tells you something about the rest of it. Newsletter ## Automating the boring half I publish one researched piece a week on putting agents to work on real sites. What I built, what broke, and the commands to check it yourself. Email address Get it weekly Free. One email a week. Unsubscribe in one click, and I do not send anything else. ## Check the domain before you trust the benchmark The MemPalace README carries a warning I have not seen on any other project in this category: > Beware of impostor sites. MemPalace has no other official websites. The only official sources are this GitHub repository and the docs at mempalaceofficial.com. Any other domain (including .tech, .net, or other .com variants) is an impostor and may distribute malware. > > MemPalace README, read 2 September 2026 ### Why that warning matters here That matters here because several of the ranking pages that surface for "best AI memory framework" are hosted on exactly those lookalike domains, and their benchmark tables are where a lot of the numbers in circulation came from. If you want MemPalace's figures, take them from `benchmarks/BENCHMARKS.md` in the repository, which commits the per-question result files and the commands to reproduce them. ![MemPalace README benchmarks section showing LongMemEval retrieval recall and which modes require no LLM](https://adityaarsharma.com/wp-content/uploads/2026/09/4550c9c8-3768-4acb-afa2-2c0866d813c8_2800x1720-scaled.png)The MemPalace README benchmarks section, read 3 September 2026. R@5, and the LLM required column is the one to read next to it. ### What the benchmark file actually says Read that file and you get a more useful picture than the headline. Raw semantic search with no heuristics and no LLM scores 96.6% R@5. A tuned hybrid pipeline, evaluated on a held-out 450 questions after tuning on 50, scores 98.4%. Adding an LLM reranker over the top 20 gets to 99% or better. The README declines to headline the 100% figure because reaching it involved inspecting specific wrong answers, which it flags as teaching to the test. A project that argues itself down from 100% is a project whose 96.6% I am inclined to believe. ## The things that will actually break Both projects document their own sharp edges. These are the ones that cost an evening. - **mem0 self-hosted auth is on by default.** The README flags it: upgrading from a pre-auth build needs `ADMIN_API_KEY` set, an admin registered through the wizard, or `AUTH_DISABLED=true` for local development only. - **mem0 hybrid search is an extra install.** Plain `pip install mem0ai` gives you semantic retrieval. BM25 keyword matching and entity extraction need `pip install mem0ai[nlp]` plus `python -m spacy download en_core_web_sm`. It is easy to benchmark the wrong configuration and conclude the retrieval is weak. - **MemPalace on Android does not work natively.** The README states that Termux is unsupported because ChromaDB and ONNX Runtime publish Linux wheels, not Android wheels. The tested path is a Debian PRoot container. - **The MemPalace GPU image is x86_64 only.** `onnxruntime-gpu` publishes no aarch64 Linux wheels, so `docker build -f Dockerfile.gpu` fails on Apple Silicon with a dependency resolution error rather than anything that names the real cause. - **MemPalace Docker on Linux hits a permissions wall.** The image runs as uid 1000 and bind mounts keep host ownership, so a `0700` directory surfaces as `PermissionError: [Errno 13]`. The README specifically warns against working around it with `--user`, because `/data` inside the image is owned by uid 1000 and no other uid can write the palace. - **First run needs network on both.** MemPalace downloads its embedding model into `/data` on the first call that needs embeddings. A slow first container is not a hung one. ## Running either one against a coding agent This is where the two projects diverge most in practice, and it is the use case most people in this search actually have. ### MemPalace with a coding agent MemPalace ships auto-save hooks for Claude Code, Codex CLI and Cursor IDE that save periodically and before context compression, plus 45 MCP tools covering palace reads and writes, knowledge graph operations, cross-wing navigation, drawer management and agent diaries. Its README carries a standing warning that Claude Code sessions expire in 30 days without those hooks wired, and points at a retention checklist for backfilling existing JSONL transcripts with `mempalace mine ~/.claude/projects/ --mode convos`. `# MemPalace, install and first index uv tool install mempalace mempalace init ~/projects/myapp mempalace mine ~/.claude/projects/ --mode convos mempalace search "why did we switch to GraphQL" mempalace wake-up` ### mem0 with a coding agent mem0 approaches the same problem through skills and a CLI. Its README documents reference skills that load SDK knowledge into an assistant's context and pipeline skills that run an end-to-end workflow in an existing repo, plus an agent signup flow that mints a working API key without an email or a dashboard. `# mem0, agent-first signup as documented in the README npm install -g @mem0/cli mem0 init --agent --agent-caller claude-code mem0 add "I am using mem0" mem0 search "am I using mem0"`Both plug into an MCP client. If you are wiring either of these into a working agent setup, the practical constraints on tool count and permissions are the same ones I hit in [running Claude Code against production WordPress](https://adityaarsharma.com/running-claude-code-against-wordpress-the-complete-setup/). Forty-five tools is not free: every tool's name, description and input schema sits in context on every request. ## What I cannot tell you I have not benchmarked these two against each other, and I am not going to publish a table that implies I did. A fair head-to-head needs the same LongMemEval split, the same metric, the same reader model, and the same hardware, run by somebody with no stake in the result. That run does not exist publicly as of 2 September 2026. ### Retrieval latency I also cannot tell you retrieval latency for either. Both projects publish figures under their own conditions. Neither published a methodology detailed enough to reproduce the other's. Anyone quoting a millisecond comparison between them is quoting two different experiments. What is checkable is everything in the table above, and the checkable part is enough to decide. It is the same discipline I applied when [a commercial security scanner reported a site clean while it was serving spam to Googlebot](https://adityaarsharma.com/the-scanner-said-clean-the-site-was-serving-spam-to-googlebot/): the vendor's headline number and the vendor's mechanism are two different claims, and only one of them can be verified from the outside. ## Pick by answering one question Does a language model need to see every item at the moment you save it? If yes, because you want structured facts extracted for a chatbot that will be queried by thousands of end users, mem0 is built for that shape and has the larger ecosystem to prove it. If no, because you are storing your own working history and you want the source text to survive the next three model generations, MemPalace's write path is the one that does not charge you per save. Then go and check the arithmetic against your own volume before you commit. Two numbers decide it: items saved per month, and average tokens per item. ## More on ai memory and mcp - [Cognee Alternatives: What You Are Actually Replacing](https://adityaarsharma.com/cognee-alternatives/)- [What Self-Hosted AI Memory Actually Costs to Run](https://adityaarsharma.com/self-hosted-ai-memory-cost/)- [MCP Server vs RAG: Which One You Actually Need](https://adityaarsharma.com/mcp-server-vs-rag/) ## Resources - mem0 repository and README: [github.com/mem0ai/mem0](https://github.com/mem0ai/mem0) - mem0 documentation: [docs.mem0.ai](https://docs.mem0.ai) - mem0 paper, Chhikara et al. 2025: [arXiv 2504.19413](https://arxiv.org/abs/2504.19413) - MemPalace repository and README: [github.com/MemPalace/mempalace](https://github.com/MemPalace/mempalace) - MemPalace official docs: [mempalaceofficial.com](https://mempalaceofficial.com) - LongMemEval, the benchmark both cite: [arXiv 2410.10813](https://arxiv.org/abs/2410.10813)