Exploratory and Assimilating Reflection: Reflective Recall Cycle for Long-term Memory

Research Question and Approach

Autonomous agents powered by Large Language Models (LLMs) often struggle with long-term memory due to their stateless architecture and limited context windows. While retrieval-augmented generation (RAG) is a standard solution, static retrievers fail to adapt to domain shifts, and existing reinforcement learning (RL) approaches for online adaptation are notoriously sample-inefficient. The authors propose Exploratory-Assimilating Reflection (EAR), a framework designed to balance immediate, query-specific retrieval with long-term model adaptation.

EAR operates through two primary mechanisms:

1. Exploratory Reflection: An iterative search process that uses an 'Explorer' module to build a rich set of evidence for a specific query, accounting for relevance, diversity, and uncertainty.
2. Assimilating Reflection: A long-term learning component that stores these exploratory experiences in a buffer and uses them to refine a global reranker, moving beyond the myopic, immediate-reward updates typical of previous RL-based methods.

EAR demonstrates significant improvements in retrieval performance across two long-term dialogue benchmarks (LongMemEval and LoCoMo). Key results include:

• Superior Retrieval: EAR achieves absolute recall gains of up to 17.9% over static baseline retrievers.
• Sample Efficiency: The framework reaches the performance levels of static baselines using 77% fewer feedback steps than existing RL-based retrieval methods, making it highly suitable for dynamic, evolving environments.
• Heterogeneous Adaptivity: EAR effectively balances different types of memory (e.g., episodic dialogue vs. semantic observational data), leading to a 13.6% improvement in downstream answering accuracy in mixed-memory scenarios.