Open Passport against Mem0, Graphiti (Zep), Supermemory and vector search, on the things a company needs from memory. On the one held-out test it did not beat Mem0 or per-department vector search. Where we lose, we say so.
Open Passport did not beat Mem0 or per-department vector search.
The pre-registered test, on data sealed before Open Passport was frozen, failed against both comparators, under both judges and both bootstrap seeds. The decisive loss is real organisational history: on the Kubernetes enhancement-proposal history Open Passport answered 143 of 300, against 205 for Mem0, 220 for per-department vector search and 221 for plain vector search. It was also 4.3 points behind Mem0 on durable facts after 24 months.
It won against Mem0 on current preferences (+12.2), house style (+37.6) and erasure (+93.1), and Mem0 used slightly fewer tokens (Open Passport / Mem0 = 1.043). Against per-department vector search it won on preferences, stale answers, rare old facts (+34.7), house style, raw company Q&A and erasure, at 0.40× its tokens. Supermemory and Graphiti (Zep) were not run at test.
Real company history (Kubernetes enhancement proposals)
N 283
vs Mem0
−20.8 [−27.2, −14.5]LOSS
vs per-dept vector
−26.5 [−32.9, −20.1]LOSS
Durable facts after 24 months of activity
N 300
vs Mem0
−4.3 [−7.0, −1.7]LOSS
vs per-dept vector
+2.7 [−1.0, 6.3]
Rare old facts after 24 months
N 300
vs Mem0
−3.0 [−6.0, 0.0]
vs per-dept vector
+34.7 [28.0, 41.0]WIN
Stale answers
N 300 · LOWER IS BETTER
vs Mem0
−1.7 [−5.0, 1.7]
vs per-dept vector
−15.3 [−20.3, −10.3]WIN
Company Q&A from raw chat and history
N 600
vs Mem0
+0.7 [−2.5, 4.0]
vs per-dept vector
+16.2 [12.0, 20.3]WIN
Company Q&A from curated notes
N 564
vs Mem0
+5.1 [3.2, 7.3]
vs per-dept vector
+7.3 [5.0, 9.8]
Current preferences, asker known
N 576
vs Mem0
+12.2 [6.9, 17.5]WIN
vs per-dept vector
+18.8 [13.7, 24.0]WIN
Invented preferences
N 127 · LOWER IS BETTER
vs Mem0
+3.1 [−1.6, 7.9]
vs per-dept vector
−6.3 [−13.4, 0.8]
Stale preferences
N 104 · LOWER IS BETTER
vs Mem0
−3.8 [−12.5, 4.8]
vs per-dept vector
−6.7 [−15.4, 1.9]
House style and taste
N 180
vs Mem0
+37.6 [33.6, 41.5]WIN
vs per-dept vector
+18.0 [14.2, 21.9]WIN
Erasure, including derived summaries (55/58 for Open Passport)
N 58
vs Mem0
+93.1 (vs 1/58)WIN
vs per-dept vector
+15.5 (vs 46/58)WIN
Tokens per answer, ratio Open Passport / comparator
N 3,103 · LOWER IS BETTER
vs Mem0
1.043× [1.023, 1.064]
vs per-dept vector
0.403× [0.395, 0.411]
Points, Open Passport − comparator [95% CI]
n
vs Mem0
vs per-dept vector
Real company history (Kubernetes enhancement proposals)
283
−20.8 [−27.2, −14.5]LOSS
−26.5 [−32.9, −20.1]LOSS
Durable facts after 24 months of activity
300
−4.3 [−7.0, −1.7]LOSS
+2.7 [−1.0, 6.3]
Rare old facts after 24 months
300
−3.0 [−6.0, 0.0]
+34.7 [28.0, 41.0]WIN
Stale answersLOWER IS BETTER
300
−1.7 [−5.0, 1.7]
−15.3 [−20.3, −10.3]WIN
Company Q&A from raw chat and history
600
+0.7 [−2.5, 4.0]
+16.2 [12.0, 20.3]WIN
Company Q&A from curated notes
564
+5.1 [3.2, 7.3]
+7.3 [5.0, 9.8]
Current preferences, asker known
576
+12.2 [6.9, 17.5]WIN
+18.8 [13.7, 24.0]WIN
Invented preferencesLOWER IS BETTER
127
+3.1 [−1.6, 7.9]
−6.3 [−13.4, 0.8]
Stale preferencesLOWER IS BETTER
104
−3.8 [−12.5, 4.8]
−6.7 [−15.4, 1.9]
House style and taste
180
+37.6 [33.6, 41.5]WIN
+18.0 [14.2, 21.9]WIN
Erasure, including derived summaries (55/58 for Open Passport)
58
+93.1 (vs 1/58)WIN
+15.5 (vs 46/58)WIN
Tokens per answer, ratio Open Passport / comparatorLOWER IS BETTER
3,103
1.043× [1.023, 1.064]
0.403× [0.395, 0.411]
Real company history, held-out
Kubernetes enhancement proposals, correct out of 300
Plain vector search221/300
Per-department vector search220/300
Mem0205/300
Open Passport143/300
Full context (32k tokens) 50, no memory 17. Tokens per answer: Open Passport 1,102, Mem0 1,454, per-department vector search 2,828.
“Win” = a pre-registered differentiator, significant after Holm correction. “Loss” = a significant loss. Untagged rows are within the margin or not registered as differentiators. Mem0 is its best-configured arm per track. Everything below this box is development-set data.
Development set
Where it leads in development.
Studies 2, 3 and 5, measured on data Open Passport was tuned on. Indicative, and they favour Open Passport.
Private, and still useful
14/16vs 0/16
0 leaks in 24 probes. With their own scoping on, the others leaked 0–1 and answered 0 shared questions.
House style
89.5vs 46.8
Answers in the company's own voice. Mem0 46.8. A written style guide in the prompt scored 91.5, level with us.
Without our date headers
90.5vs 78.9
Real company history with no Open Passport formatting. Mem0 78.9; per-department vector search 85.0, a tie.
Junk writes stored
4/39vs 24/39
Junk and poison writes kept after raw ingest. Mem0 kept 24, Graphiti (Zep) 6. Lower is better.
Development results
Every development number, one table.
% correct unless noted. Open Passport is highlighted. A dash means the system was not run on that test.
Company Q&A from curated notes
STUDY 2
Open Passport
97.3
Mem0
91.0
Graphiti (Zep)
62.8
Supermemory
87.2
Plain vector search
95.7tie
Company Q&A from raw chat and history
STUDY 2
Open Passport
86.0
Mem0
84.0tie
Graphiti (Zep)
74.5
Supermemory
79.5tie
Plain vector search
49.5
Current value after 1–4 changes, curated notes
STUDY 2
Open Passport
32/32
Mem0
29/32
Graphiti (Zep)
8/32
Supermemory
23/32
Plain vector search
30/32
“As of date X, what was the value?”, curated notes
STUDY 2
Open Passport
18/20
Mem0
11/20
Graphiti (Zep)
1/20
Supermemory
11/20
Plain vector search
19/20
“What changed since date X, and who changed it?”, curated notes
STUDY 2
Open Passport
19/20
Mem0
20/20
Graphiti (Zep)
2/20
Supermemory
14/20
Plain vector search
20/20
Cross-department leaks, each system with its own per-department scoping on
STUDY 2 · LOWER IS BETTER
Open Passport
0/24
Mem0
1/24
Graphiti (Zep)
0/24
Supermemory
1/24
Plain vector search
1/24
Useful cross-department answers, same scoped setup
STUDY 2
Open Passport
14/16
Mem0
0/16
Graphiti (Zep)
0/16
Supermemory
0/16
Plain vector search
0/16
Cross-department leaks with no access restriction (Open Passport is always scoped)
STUDY 2 · LOWER IS BETTER
Mem0
7/24
Graphiti (Zep)
6/24
Supermemory
10/24
Plain vector search
7/24
Junk writes stored after raw ingest
STUDY 2 · LOWER IS BETTER
Open Passport
4/39
Mem0
24/39
Graphiti (Zep)
6/39
Tokens per answer, median, curated Q&A
STUDY 2 · LOWER IS BETTER
Open Passport
1,190
Mem0
704
Graphiti (Zep)
1,974
Supermemory
9,772
Plain vector search
2,065
Company Q&A from curated notes (later version)
STUDY 3
Open Passport
97.9
Mem0
91.0
Graphiti (Zep)
62.8
Supermemory
87.2
Plain vector search
95.7tie
House style: answers in the company's own voice
STUDY 3
Open Passport
89.5
Mem0
46.8
Graphiti (Zep)
47.6
Supermemory
79.1
Plain vector search
52.0
Durable facts after 24 months of activity
STUDY 3
Open Passport
96
Mem0
95tie
Plain vector search
83
Per-department vector search
84
Rare facts after 24 months, mentioned once
STUDY 3
Open Passport
82
Mem0
88tie
Plain vector search
50
Per-department vector search
50
Answers using a value that had since changed, %
STUDY 3 · LOWER IS BETTER
Open Passport
6.7
Mem0
1.7tie
Plain vector search
26.7
Per-department vector search
26.7
Real company history (Python enhancement proposals)
STUDY 3
Open Passport
91.8
Mem0
84.4
Plain vector search
92.5tie
Per-department vector search
97.3
Python history, standard wording (newest research version)
STUDY 5
Open Passport
88.4
Mem0
79.6
Per-department vector search
91.2
Python history without Open Passport's own date headers (newest research version)
STUDY 5
Open Passport
90.5
Mem0
78.9
Per-department vector search
85.0tie
Worst score across 4 reworded and restyled copies of the history (newest research version)
STUDY 5
Open Passport
86.4
Mem0
78.9
Per-department vector search
85.0tie
Erasure: forget a fact on request, nothing left behind
STUDY 5
Open Passport
32/32
Tokens per answer across the 4 copies (newest research version)
STUDY 5 · LOWER IS BETTER
Open Passport
1,089–1,242
Mem0
1,231–1,286
Per-department vector search
2,645–2,838
Test
Open Passport
Mem0
Graphiti (Zep)
Supermemory
Plain vector search
Per-department vector search
Company Q&A from curated notesSTUDY 2
97.3
91.0
62.8
87.2
95.7tie
–
Company Q&A from raw chat and historySTUDY 2
86.0
84.0tie
74.5
79.5tie
49.5
–
Current value after 1–4 changes, curated notesSTUDY 2
32/32
29/32
8/32
23/32
30/32
–
“As of date X, what was the value?”, curated notesSTUDY 2
18/20
11/20
1/20
11/20
19/20
–
“What changed since date X, and who changed it?”, curated notesSTUDY 2
19/20
20/20
2/20
14/20
20/20
–
Cross-department leaks, each system with its own per-department scoping onSTUDY 2 · LOWER IS BETTER
0/24
1/24
0/24
1/24
1/24
–
Useful cross-department answers, same scoped setupSTUDY 2
14/16
0/16
0/16
0/16
0/16
–
Cross-department leaks with no access restriction (Open Passport is always scoped)STUDY 2 · LOWER IS BETTER
–
7/24
6/24
10/24
7/24
–
Junk writes stored after raw ingestSTUDY 2 · LOWER IS BETTER
4/39
24/39
6/39
–
–
–
Tokens per answer, median, curated Q&ASTUDY 2 · LOWER IS BETTER
1,190
704
1,974
9,772
2,065
–
Company Q&A from curated notes (later version)STUDY 3
97.9
91.0
62.8
87.2
95.7tie
–
House style: answers in the company's own voiceSTUDY 3
89.5
46.8
47.6
79.1
52.0
–
Durable facts after 24 months of activitySTUDY 3
96
95tie
–
–
83
84
Rare facts after 24 months, mentioned onceSTUDY 3
82
88tie
–
–
50
50
Answers using a value that had since changed, %STUDY 3 · LOWER IS BETTER
6.7
1.7tie
–
–
26.7
26.7
Real company history (Python enhancement proposals)STUDY 3
91.8
84.4
–
–
92.5tie
97.3
Python history, standard wording (newest research version)STUDY 5
88.4
79.6
–
–
–
91.2
Python history without Open Passport's own date headers (newest research version)STUDY 5
90.5
78.9
–
–
–
85.0tie
Worst score across 4 reworded and restyled copies of the history (newest research version)STUDY 5
86.4
78.9
–
–
–
85.0tie
Erasure: forget a fact on request, nothing left behindSTUDY 5
32/32
–
–
–
–
–
Tokens per answer across the 4 copies (newest research version)STUDY 5 · LOWER IS BETTER
1,089–1,242
1,231–1,286
–
–
–
2,645–2,838
“Tie” means the confidence interval of the difference includes zero. Rows come from different studies and versions; compare within a row.1
Side by side · development set
The key comparisons.
Company Q&A from curated notes
% correct, study 2
Open Passport97.3
Plain vector search95.7 (tie)
Mem091.0
Supermemory87.2
Graphiti (Zep)62.8
House style
% of answers in the company’s own voice, study 3
Open Passport89.5
Supermemory79.1
Plain vector search52.0
Graphiti (Zep)47.6
Mem046.8
History without our date headers
% correct, newest research version, study 5
Open Passport90.5
Per-department vector search85.0 (tie)
Mem078.9
Useful cross-department answers, with 0–1 leaks
right answers out of 16, every system scoped by department, study 2
Open Passport14/16
Mem00/16
Graphiti (Zep)0/16
Supermemory0/16
Plain vector search0/16
Tokens per answer
newest research version, upper end of the range, study 5· LOWER IS BETTER
Open Passport1,089–1,242
Mem01,231–1,286
Per-department vector search2,645–2,838
Open PassportMem0Graphiti (Zep)SupermemoryPlain vector searchPer-department vector search
Honest numbers
Where we don’t win yet.
Benchmarks we lose, or do not clearly win, held-out results first. We are working on each.
Real company history · held-out
Kubernetes enhancement-proposal history, unseen until the test: 143 of 300 right, against 205 for Mem0, 220 for per-department vector search and 221 for plain vector search. That is 20.8 points behind Mem0 and 26.5 behind per-department vector search, and it decided the test.
Durable facts after 2 years · held-out
After 24 months of company activity, 4.3 points behind Mem0 (95% CI −7.0 to −1.7).
Tokens · held-out
Mem0 used slightly fewer tokens per answer: Open Passport used 1.043× as many.
Preferences, asker unknown · held-out
When the system is not told who is asking: per-department vector search 27.1, Open Passport 19.8, Mem0 12.3.
Preferences against full context · held-out
Putting the whole chat history in the prompt beat Open Passport on current preferences by 11.1 points.
Real company history · development
On Python enhancement-proposal history, per-department vector search was ahead too: 97.3 against 91.8 in study 3, and 91.2 against 88.4 in study 5.
Rare facts after 2 years · development
A fact mentioned once, asked about two years later: 82 for Open Passport, 88 for Mem0.
Answering with a stale fact · development
How often the answer used a value that had since changed: 6.7% for Open Passport, 1.7% for Mem0.
Date questions on curated notes · development
Plain vector search matched or beat us: 19/20 against 18/20 on “as of date X”, and 20/20 against 19/20 on “what changed since date X” (Mem0 also 20/20).
“What changed since X?” on real history · development
Weak for everyone: Open Passport 4–10 of 20, the others 4–11 of 20.
Leaks against scoped competitors · development
With their own per-department scoping on, the others leaked 0–1 of 24, level with us. The difference is that they then answered 0 of 16 cross-team questions; Open Passport answered 14.
House style vs a style guide · development
Pasting a plain written style guide into the prompt scored 91.5, level with Open Passport. On the held-out test our house-style wins are over Mem0 and per-department vector search only; a style guide was not compared there.
Fresh conflicting writes · development
When an app wrote a newer, correct value, Open Passport held it for review and served 0 of 12; Mem0 served 7 of 12.
Forgetting · development
Importance-based forgetting archived nothing in 24 months of activity.
LoCoMo
An academic benchmark of long personal chats, not company memory. We are about 22 points behind Mem0.
Tokens in study 2 · development
Open Passport used a median 1,190 tokens per answer against Mem0’s 704. The newest research version is about level with Mem0 (study 5).
Why a new benchmark
What makes a company-memory benchmark different.
Personal-chat benchmarks test one person talking to one assistant. A company is different.
Many people and departments
Sales, support and engineering each know different things.
Decisions and provenance
What was decided, by whom, and where it came from.
Facts that change
Prices, owners and policies move. The answer depends on the date.
House style
Answers should sound like the company, not like a chatbot.
Privacy between departments
One department's confidential facts must not reach another.
Forgetting on request
Erase a fact, and nothing of it is left to resurface.
Tokens per answer
Memory is loaded on every call, so its size is a running cost.
Methodology
How we measured.
01The held-out test (study 4) ran once, on 2026-10-26, on data sealed before Open Passport was frozen. Its verdict is negative: Open Passport did not beat Mem0 or per-department vector search under the pre-registered rule.
02Every other number on this page is a development-set result from pre-registered studies 2, 3 and 5: measured on data Open Passport was tuned on, with few companies and no multiplicity correction. Read them as indicative; they favour Open Passport.
03Scores are % correct, with paired bootstrap confidence intervals (10,000 resamples; on the held-out test stratified by cluster, with Holm correction, a second judge and a second seed). “Tie” means the interval for the difference includes zero.
04Answers are graded against an answer key plus judges.
05Different rows come from different studies and Open Passport research versions, so compare within a row, not across rows. Rows marked “newest research version” come from a version that has not been held-out tested.
06Each competitor ran with its documented defaults and hooks, and every system shared the same answering model, embedder and answer prompt.
Notes
1 Versions tested: Mem0 2.2.1 (open source), best configuration per test; Graphiti 0.31.2 with FalkorDB, the open-source engine behind Zep; Supermemory local 0.0.9; Plain vector search one embedding index, 400-token chunks (scoped per department in the leak rows); Per-department vector search one embedding index per department plus a company index, 200-token chunks.
2 Token ranges for the newest research version span the four copies of the Python history (study 5). It has not been held-out tested; the held-out test used the version chosen at the end of study 4's development phase.
Give every AI your business uses the same memory.
The hosted Passport is in private beta. The standard is open today.