MemArena benchmark shows memory backend matters more than model size
Researchers Jiadong Zhang and Xiaosong Ma introduce MemArena, a benchmark for agentic personal memory assistants meant to run on-device with open-weight models, handling private interpersonal conversations locally rather than in the cloud. The benchmark targets a gap the authors identify in existing memory benchmarks, which under-test the combination of activity-dense interaction, an ego-centric (first-person) perspective, and coherent multi-session worlds.
MemArena is built around a single simulated world generated by the authors' own MASim agent simulator: 50 agents interacting over 15 days, producing 10.3 million dialog-text tokens in total and 24,100 text-only ego-observed tokens per agent per day. From that interaction history, the benchmark co-generates ground truth across six evaluation dimensions covering recall, reasoning, and trustworthiness.
Using this setup, the authors evaluate five open-weight reader models against five memory backends: plain (vanilla) context, BM25-based retrieval-augmented generation (BM25-RAG), an oracle retrieval baseline, Memobase, and MemSearch. Three findings stand out. First, the choice of memory backend affects content accuracy more than scaling up the reader model: for the Qwen3-0.6B reader, switching the backend from Memobase to MemSearch produces accuracy gains of +32.5 and +19.2 percentage points on two metrics, larger than the +10.6 and +6.8 percentage-point gains obtained by scaling the MemSearch reader itself. Second, permission-aware access control fails across the board: the oracle-retrieval backend leaks private information heavily, while the other backends err the opposite way, staying too cautious to disclose information they should share. Third, search latency is a meaningful bottleneck only for very small reader models; measured on a Spark GB10 edge node, memory search adds a fixed 87, 7, and 48 milliseconds for BM25-RAG, Memobase, and MemSearch respectively, a small share of time-to-first-token for most reader-backend combinations.
The abstract reports no absolute accuracy scores for any backend or reader, only percentage-point differences between configurations, and it does not explain what Memobase or MemSearch are as systems beyond naming them as backends under test. The authors say the code, the MASim simulator, and a further MemArena-L benchmark will be released once the paper is accepted, with no date given.
Key facts
- MemArena simulates 50 agents over 15 days, generating 10.3M dialog-text tokens and 24.1K ego-observed tokens per agent per day.
- Ground truth spans six recall, reasoning, and trustworthiness dimensions, tested against five open-weight readers and five memory backends: Vanilla context, BM25-RAG, Oracle, Memobase, and MemSearch.
- At the Qwen3-0.6B reader, switching backend from Memobase to MemSearch gains +32.5/+19.2 percentage points, more than the +10.6/+6.8 pp gained by scaling the MemSearch reader itself.
- Permission-aware access fails universally: Oracle retrieval leaks private data heavily, while the other backends are too cautious to disclose information they should.
- On a Spark GB10 edge node, memory search adds a fixed 87/7/48 ms latency for BM25-RAG/Memobase/MemSearch, a small fraction of time-to-first-token except at very small reader sizes.
Why it matters
On-device personal memory assistants need to handle private conversations locally with open-weight models, but existing benchmarks under-test the mix of activity-dense interaction, first-person perspective, and coherent multi-session history that real use requires. MemArena is built to close that gap, using a purpose-built agent simulator, MASim, to generate one consistent 15-day, 50-agent world rather than stitching together shorter or less realistic sessions.
Who it affects
Developers building on-device or edge agentic systems with memory, researchers evaluating memory architectures for open-weight models, and hardware teams targeting edge nodes such as the Spark GB10 used in the paper's latency measurements.
How to use it
The paper is available now on arXiv. The authors say the code, the MASim simulator, and the MemArena-L benchmark will be released once the paper is accepted, but give no release date or venue.
How solid is it
The paper runs 48 pages with 6 figures and evaluates five reader models against five memory backends on a purpose-built 50-agent, 15-day simulated benchmark scored across six dimensions, giving the comparison some breadth. The abstract reports only relative accuracy gains between configurations, not absolute accuracy scores for any backend or reader, and as an arXiv preprint it has not yet been peer reviewed or accepted; release of the code and benchmark data is still pending acceptance.
Risks and caveats
The paper's own headline finding is itself a caveat: permission-aware access fails universally across every backend tested, with the oracle-retrieval baseline leaking private information heavily and the remaining backends erring toward withholding information they should disclose, meaning no tested configuration got this tradeoff right. The abstract also does not name the authors' institutions, does not define what Memobase or MemSearch are as systems, and does not specify what the Spark GB10 edge node is beyond being the hardware used for the latency measurements.
“Permission-aware access fails universally, with Oracle leaking heavily and other backends too timid to disclose.”
— MemArena paper (Jiadong Zhang and Xiaosong Ma)