HiVe-Mem gives multimodal agents a two-way memory and cuts tokens by about 90%

The paper, listed on Hugging Face Papers, starts from a practical problem. Multimodal agents build up longer and longer interaction histories, and for long-term interaction they need to find and use the right piece of the past. The authors argue that existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations. That can miss relevant memories and ignore the fact that different queries need different kinds of evidence.
Their answer is HiVe-Mem, a multimodal memory framework that interleaves two steps. The name comes from them: Horizontal memory expansion and Vertical evidence composition.
The horizontal step is about coverage. HiVe-Mem organizes memory episodes into a relational graph, using textual and visual anchors. Semantic retrieval first finds seed memories. The system then expands those seeds with related neighbors in the graph, to recover complementary information that a plain similarity search would not return.
The vertical step is about the form of the evidence. Each retrieved memory keeps several evidence forms: compact summaries, captions, grounded visual views, raw images and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory. This makes the use of heterogeneous multimodal information query-adaptive, so the agent does not have to pass everything, or only a fixed representation, to the model.
The authors say this two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories.
On results, the abstract reports experiments across three multimodal memory benchmarks. HiVe-Mem consistently improves task performance across base models, while reducing token consumption by about 90% compared with the most efficient competing multimodal baselines. The abstract gives no per-benchmark scores, and it does not name the benchmarks or the base models.
Key facts
- HiVe-Mem is a multimodal memory framework for agents with long interaction histories; it interleaves Horizontal memory expansion with Vertical evidence composition.
- Horizontal step: memory episodes are organized into a relational graph using textual and visual anchors, and semantically retrieved seeds are expanded with related neighbors.
- Vertical step: each memory keeps summaries, captions, grounded visual views, raw images and interaction history, and a lightweight trainable selector picks a subset per memory for each query.
- Across three multimodal memory benchmarks, the authors report consistent performance gains across base models.
- Token consumption is reduced by about 90% compared with the most efficient competing multimodal baselines.
Why it matters
Agents that run for a long time collect large histories of text and images, and what they can recall limits what they can do. The paper targets two weaknesses it ascribes to current memory systems: semantic retrieval over compressed memories can miss relevant items, and fixed predefined representations ignore that different queries need different evidence. HiVe-Mem tries to fix both at once, by widening what is retrieved and by choosing the form of each retrieved item. The headline efficiency claim is about 90% lower token consumption against the most efficient competing multimodal baselines.
Who it affects
Mainly people building multimodal agents that must remember long interaction histories, and researchers working on agent memory. The abstract does not name any lab, product or deployment.
How to use it
The abstract describes a design rather than a release. Its building blocks are a relational graph of memory episodes with textual and visual anchors, neighbor expansion from retrieved seeds, and a lightweight trainable selector that chooses among summaries, captions, grounded visual views, raw images and interaction history for each memory. Teams working on agent memory could read the full paper for how these parts are implemented and trained. The abstract does not mention a code or model release.
How solid is it
This is a preprint abstract. The claims are the authors' own: consistent gains across base models on three multimodal memory benchmarks and about 90% lower token use than the most efficient competing multimodal baselines. The abstract does not name the benchmarks or the base models, and it gives no per-benchmark accuracy numbers, so the size of the gains cannot be judged from it. The 90% figure is stated as approximate and only against the most efficient competing multimodal baselines, not against all baselines.
Risks and caveats
The performance gains are described only as consistent, with no figures. The token saving is relative to the most efficient competitors, and it is not clear from the abstract how it varies across the three benchmarks. The size of the selector and its training details are not given beyond 'lightweight trainable'. Independent replication is not mentioned.