MEMPROBE: Probing Long-Term Agent Memory via Hidden User-State Recovery

Long-term memory is a critical component for LLM agents, intended to allow them to build a durable, evolving understanding of the users they serve. Current research typically evaluates these memory systems by measuring downstream performance—such as task success, response quality, or personalization accuracy. However, this approach is fundamentally flawed because a high task-success rate does not necessarily mean the agent has formed a reliable model of the user. An agent might complete a task through lucky guesses, context-window information, or simple pattern matching, leaving the actual memory artifact unexamined and unaudited.

MEMPROBE shifts the evaluation paradigm from behavioral outcomes to direct artifact auditing. The researchers developed a benchmark featuring 50 simulated users, each with a hidden, taxonomy-anchored bank of 31 user-state dimensions (covering skills, knowledge, episodic events, self-models, and preferences). During interaction, these agents perform leak-controlled tasks that naturally elicit evidence about the user without explicitly naming the hidden state. After the interaction, MEMPROBE reconstructs the user's hidden state from the agent's memory store using two distinct modes: a full-store dump and a standard top-k retrieval interface. This dual-mode approach allows researchers to pinpoint whether a failure occurred because the information was never written to memory or because it was written but remains unreachable.

Testing five state-of-the-art memory systems, the study highlights three major insights. First, task success is a saturated metric that fails to distinguish between memory-equipped agents and memoryless baselines, confirming that behavioral scores are insufficient for evaluating memory quality. Second, there is a significant gap in retrieval-aware consolidation; while agents often capture raw interaction data, they struggle to distill this into durable, reusable claims. Finally, the study identifies episodic and relational memory as the primary frontier. While agents are becoming adept at storing stable preferences, they struggle to link one-off experiences to the context and consequences that give them meaning, which is essential for an agent to genuinely know a user.