As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce \textbf{\hivemem}, a multimodal memory framework that interleaves \textbf{\textsc{H}}or\textbf{\textsc{i}}zontal Memory Expansion with \textbf{\textsc{Ve}}rtical Evidence Composition. For horizontal memory expansion, \hivemem organizes memory episodes into a relational graph using textual and visual anchors, expanding semantically retrieved seeds with related neighbors to recover complementary information. For vertical evidence composition, each retrieved memory retains multiple evidence forms, including compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory, enabling query-adaptive utilization of heterogeneous multimodal information. This two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories. Experiments across three multimodal memory benchmarks show that \hivemem consistently improves task performance across base models while reducing token consumption by about 90\% compared with the most efficient competing multimodal baselines.
Sam: Remembering more does not have to mean reading more. A new agent-memory framework improved answers while using about ninety percent fewer tokens than competing multimodal systems.
Alex: That sounds like a useful trade, but what are these agents actually remembering?
Sam: Past conversations containing text and images. The framework is called HIVE-MEM, from Haozhen Zhang, Bohan Liu, and colleagues, including researchers at Nanyang Technological University. It is a method paper presented as an arXiv preprint.
Alex: So this matters if your agent needs something from an old conversation, but cannot keep rereading its entire history.
Sam: And if the answer depends on a photograph, a short written memory might not preserve it. The authors separate finding useful past interactions from deciding what to show the answering model. We’ll follow that distinction, examine the benchmark gains, and ask how much the comparisons really establish.
Alex: Start with the existing approaches. What do they lose when they turn an interaction into a memory?
Sam: The authors identify two problems. Semantic retrieval, meaning matching a question to similar stored descriptions, can miss related interactions. And once a memory is retrieved, systems often expose a predefined representation, even though different questions need different details.
Alex: A summary could preserve the topic but lose the detail that a later question happens to need.
Sam: That is the concern. Existing multimodal systems, which handle text and images, already have sophisticated storage. MIRIX uses specialized memory types, while MemVerse builds hierarchical knowledge structures. HIVE-MEM focuses on how those stored experiences get used when a question arrives.
Alex: What does it store that lets it make a different decision later?
Sam: It divides the history into short, contiguous episodes. Each episode keeps its original text and images, alongside a summary and image captions. It also stores a grounded visual view: a version that localizes context-relevant image regions and suppresses distracting background.
Alex: So the summary is not replacing the original material. It is another way into it.
Sam: The episode also gets anchors: short cues extracted from text and images. These might name an entity, an event, an object, or a scene. The anchors help connect episodes that belong together, even when their summaries do not closely match the same question.
Alex: Can we make that concrete with the paper’s example? I want to know what a summary search would miss.
Sam: The illustration asks, “What color was Mei’s bag from Kyoto?” Some memories concern shopping in Kyoto with Mei. Another describes a red canvas tote bag. The graph can connect trip-related memories with bag-related memories, helping bring the color evidence into reach.
Alex: The useful link might cross the text-image boundary, then. A bag mentioned in one conversation could appear in another episode’s image.
Sam: That is one of the connections the method explicitly supports. Shared textual cues connect episodes, and textual cues can also match visual cues. The graph’s links do not require a predefined relationship such as “owns” or “visited.”
Alex: Shared cues can also be generic. If everything contains a person or a room, following those links could just add clutter.
Sam: The authors downweight anchors that appear throughout the memory bank. More distinctive anchors count for more. At query time, candidates also score better when their connecting anchors are relevant to the question, and the graph limits each episode’s connections.
Alex: That handles which memories to retrieve. What happens when those memories arrive with summaries, captions, photographs, and original conversations attached?
Sam: That is the second decision: evidence composition, or choosing what each memory contributes. A lightweight selector considers the question and an episode’s summary. It can independently include or exclude each available evidence form, rather than choosing one fixed package for every episode.
Alex: Does that selector actually inspect the photograph before deciding whether the answering model needs it?
Sam: No. Its input is the question and the compact summary, represented as numerical embeddings. That keeps selection lightweight, but the decision is based on those compact descriptions, not a fresh inspection of every image.
Alex: How do the authors teach it which package will be useful, rather than simply asking it to guess?
Sam: They use reinforcement learning: training from rewards tied to downstream answers. The reward can balance answer quality against the tokens consumed by selected evidence. Tokens are the units counted in the model’s input and output.
Alex: Then the obvious possibility is that the savings just come from heavily penalizing expensive evidence. Was that the main experimental setting?
Sam: No. The default experiments set the explicit token penalty to zero. They still report large savings. Separately, increasing that penalty shifts the selector toward more compact evidence, giving different tradeoffs between answer quality and consumption.
Alex: Let’s turn to the evidence. How broad was the evaluation, and were the other systems allowed the same retrieval count?
Sam: They evaluated three multimodal memory benchmarks, using both a Qwen vision-language model and GPT-5-mini. The comparison held the retrieved-memory budget at seven. HIVE-MEM split that allowance between direct search and graph expansion, while the baselines used direct retrieval.
Alex: What tells us the answers improved, rather than merely becoming shorter and cheaper?
Sam: They measured answer overlap with reference answers and used a language model to judge answer quality. On the Qwen backbone, HIVE-MEM’s average judge score was about fifty-eight. The strongest competing multimodal average was about forty-eight. Its average answer-overlap score also led the multimodal methods.
Alex: That is a model-based evaluation alongside answer overlap, not a report of human users finding the agent more useful.
Sam: Correct. The gains are benchmark results. Both backbones improved on average, and the selector transferred to GPT-5-mini without retraining. The headline token savings compare against the most token-efficient competing multimodal baseline, not against the cheaper text-only systems, which operate under a different modality setting.
Alex: Which component seems to carry the result? Finding extra related episodes, or changing what the model sees?
Sam: Removing graph expansion modestly lowered the judge score, even when the retrieval count stayed fixed. Removing visual anchors or their specificity weighting also hurt. The much larger loss came when every retrieved episode contributed only its summary: the average judge score fell to about forty-four.
Alex: But that last comparison removes original evidence as well as adaptive selection. Does it show that learning to choose is better than a fixed, richer package?
Sam: Not by itself. That ablation shows that summaries alone are insufficient here. Removing original text or raw images individually also hurts. My reading is that these results support preserving complementary evidence more directly than they isolate the benefit of learned adaptation over every fixed alternative.
Alex: Do the selector’s actual choices at least match the information the questions require?
Sam: They show useful patterns. Text-dominant questions almost never select images or grounded views. Visual questions select them more often, but still not for every retrieved memory. Original conversation text remains a strong backbone, so this is not simply replacing text with pictures whenever vision is involved.
Alex: Who should read the full document, and where would you send them first?
Sam: Researchers building long-term memory for text-and-image agents should start with the methodology, especially the episode graph and per-memory selector. Then read the ablation study, watching the difference between removing adaptive composition and restricting evidence to summaries. For deployment decisions, also inspect the runtime and budget analyses; these benchmarks do not establish performance in a live, continuously changing service.
Alex: And what should someone outside agent-memory research carry away?
Sam: Retrieving the right memory and showing the right evidence are separate decisions. This study suggests that treating them separately can improve answers without making the model read more.
Alex: Thanks for listening.