Haozhen Zhang, Bohan Liu, Jianzhu Bao, Qingyuan Liu, Haodong Yue, Xiao Li, Zi Pong Lim, Wenya Wang
8 min
Abstract
As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce \textbf{\hivemem}, a multimodal memory framework that interleaves \textbf{\textsc{H}}or\textbf{\textsc{i}}zontal Memory Expansion with \textbf{\textsc{Ve}}rtical Evidence Composition. For horizontal memory expansion, \hivemem organizes memory episodes into a relational graph using textual and visual anchors, expanding semantically retrieved seeds with related neighbors to recover complementary information. For vertical evidence composition, each retrieved memory retains multiple evidence forms, including compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory, enabling query-adaptive utilization of heterogeneous multimodal information. This two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories. Experiments across three multimodal memory benchmarks show that \hivemem consistently improves task performance across base models while reducing token consumption by about 90\% compared with the most efficient competing multimodal baselines.
Sam: The illustration asks, “What color was Mei’s bag from Kyoto?” Some memories concern shopping in Kyoto with Mei. Another describes a red canvas tote bag. The graph can connect trip-related memories with bag-related memories, helping bring the color evidence into reach.
Alex: The useful link might cross the text-image boundary, then. A bag mentioned in one conversation could appear in another episode’s image.
Sam: That is one of the connections the method explicitly supports. Shared textual cues connect episodes, and textual cues can also match visual cues. The graph’s links do not require a predefined relationship such as “owns” or “visited.”
Alex: Shared cues can also be generic. If everything contains a person or a room, following those links could just add clutter.
Sam: The authors downweight anchors that appear throughout the memory bank. More distinctive anchors count for more. At query time, candidates also score better when their connecting anchors are relevant to the question, and the graph limits each episode’s connections.
Alex: That handles which memories to retrieve. What happens when those memories arrive with summaries, captions, photographs, and original conversations attached?
Sam: That is the second decision: evidence composition, or choosing what each memory contributes. A lightweight selector considers the question and an episode’s summary. It can independently include or exclude each available evidence form, rather than choosing one fixed package for every episode.
Alex: Does that selector actually inspect the photograph before deciding whether the answering model needs it?
Sam: No. Its input is the question and the compact summary, represented as numerical embeddings. That keeps selection lightweight, but the decision is based on those compact descriptions, not a fresh inspection of every image.
Alex: How do the authors teach it which package will be useful, rather than simply asking it to guess?
Sam: They use reinforcement learning: training from rewards tied to downstream answers. The reward can balance answer quality against the tokens consumed by selected evidence. Tokens are the units counted in the model’s input and output.
Alex: Then the obvious possibility is that the savings just come from heavily penalizing expensive evidence. Was that the main experimental setting?
Sam: No. The default experiments set the explicit token penalty to zero. They still report large savings. Separately, increasing that penalty shifts the selector toward more compact evidence, giving different tradeoffs between answer quality and consumption.
Alex: Let’s turn to the evidence. How broad was the evaluation, and were the other systems allowed the same retrieval count?
Sam: They evaluated three multimodal memory benchmarks, using both a Qwen vision-language model and GPT-5-mini. The comparison held the retrieved-memory budget at seven. HIVE-MEM split that allowance between direct search and graph expansion, while the baselines used direct retrieval.
Alex: What tells us the answers improved, rather than merely becoming shorter and cheaper?
Sam: They measured answer overlap with reference answers and used a language model to judge answer quality. On the Qwen backbone, HIVE-MEM’s average judge score was about fifty-eight. The strongest competing multimodal average was about forty-eight. Its average answer-overlap score also led the multimodal methods.
Alex: That is a model-based evaluation alongside answer overlap, not a report of human users finding the agent more useful.
Sam: Correct. The gains are benchmark results. Both backbones improved on average, and the selector transferred to GPT-5-mini without retraining. The headline token savings compare against the most token-efficient competing multimodal baseline, not against the cheaper text-only systems, which operate under a different modality setting.
Alex: Which component seems to carry the result? Finding extra related episodes, or changing what the model sees?
Sam: Removing graph expansion modestly lowered the judge score, even when the retrieval count stayed fixed. Removing visual anchors or their specificity weighting also hurt. The much larger loss came when every retrieved episode contributed only its summary: the average judge score fell to about forty-four.
Alex: But that last comparison removes original evidence as well as adaptive selection. Does it show that learning to choose is better than a fixed, richer package?
Sam: Not by itself. That ablation shows that summaries alone are insufficient here. Removing original text or raw images individually also hurts. My reading is that these results support preserving complementary evidence more directly than they isolate the benefit of learned adaptation over every fixed alternative.
Alex: Do the selector’s actual choices at least match the information the questions require?
Sam: They show useful patterns. Text-dominant questions almost never select images or grounded views. Visual questions select them more often, but still not for every retrieved memory. Original conversation text remains a strong backbone, so this is not simply replacing text with pictures whenever vision is involved.
Alex: Who should read the full document, and where would you send them first?
Sam: Researchers building long-term memory for text-and-image agents should start with the methodology, especially the episode graph and per-memory selector. Then read the ablation study, watching the difference between removing adaptive composition and restricting evidence to summaries. For deployment decisions, also inspect the runtime and budget analyses; these benchmarks do not establish performance in a live, continuously changing service.
Alex: And what should someone outside agent-memory research carry away?
Sam: Retrieving the right memory and showing the right evidence are separate decisions. This study suggests that treating them separately can improve answers without making the model read more.
Alex: Thanks for listening.