ContextWeave: A Real-World Workflow Benchmark

As language agents transition from isolated, prompt-based tasks to complex, stateful workflows, the ability to retain and utilize long-term experience becomes critical. The authors investigate whether current memory systems actually improve agent performance in realistic, multi-session office environments, moving beyond simple retrieval-based question answering to evaluate how memory influences downstream task execution.

The authors introduce ContextWeave, a longitudinal benchmark derived from the multi-month work histories of 14 participants. The dataset comprises 1,005 executable tasks, including 568 core evaluation tasks, reconstructed into containerized environments with privacy-preserved artifacts. The benchmark evaluates agents by comparing their performance on target tasks with and without access to preceding workflow histories. It measures success through two primary metrics: Workspace Score (task completion quality) and Preference Score (alignment with user-specific work practices), while providing diagnostic metrics for relevance, continuity, and robustness to misleading recall.

Experimental results across six memory configurations and five base models show that access to prior experience consistently improves both Workspace and Preference scores. The study reveals that memory is most effective when it supports workflow continuation rather than just acting as a retrieval mechanism. Specifically, memory systems that provide concrete, actionable, in-context experience reduce redundant exploration more effectively than those relying on compact summaries. However, the authors also observe that these richer memory representations can be more susceptible to errors if the recalled information is misleading, highlighting a trade-off between the depth of experience and the reliability of its application.