Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents

Research Question and Approach

This paper investigates how different types of retrieved memory influence the quality of responses in Retrieval-Augmented Generation (RAG) conversational agents. While previous research has focused heavily on storage and retrieval mechanisms, this study shifts the focus to the functional roles of individual memory pieces. The authors introduce a fine-grained taxonomy to classify memory into five distinct roles—answer, clarifying, enriching, distracting, and irrelevant—and develop a user-centric evaluation framework to measure response quality beyond simple reference-based metrics.

Methodology

To test their hypothesis, the researchers conducted experiments using two long-term, multi-session conversational datasets (LongMemEval-m and Long-MT-Bench+). They evaluated three frontier LLMs (GPT-4o, Gemini-2.0-Flash, and Llama-3.3-70B) across varying context sizes (from 0 to 100 retrieved pieces). The study employed an LLM-as-a-judge (GPT-o3) to classify memory roles and evaluate response quality across dimensions such as accuracy, relevance, and informativeness, ensuring the evaluation was sensitive to user intent and conversational context.

Key Findings

• Memory Selectivity: Frontier LLMs demonstrate a capacity for nuanced, selective use of contextual information, but performance often follows a non-linear trend where increasing the retrieval size beyond a certain point leads to performance degradation.
• Role-Specific Impacts: Clarifying memory is essential for improving factual accuracy and grounding, often enabling high-quality responses even in isolation. Conversely, irrelevant memory acts as noise that reduces topic relevance and weakens the agent's awareness of conversational constraints.
• Divergence from Prior Work: The study finds that the negative impact of irrelevant memory on frontier LLMs contradicts earlier findings on smaller-scale models, highlighting the need for role-aware retrieval strategies in modern conversational systems.