Ningyu Zhang
5 min
Abstract
The next generation of AI agents is increasingly moving beyond systems that answer isolated questions toward persistent personal assistants that can understand, remember, and continuously learn from users' experiences. Such assistants require long-term memory to accumulate and leverage user-specific experiences over time, yet existing benchmarks remain inadequate for realistic mobile settings, where experiences are heterogeneous, multimodal, evolving, and deeply personal. We introduce MobileMem, a benchmark and framework for studying on-device long-term memory, grounded in a year-scale collection of mobile experiences. MobileMem employs a knowledge-grounded synthesis pipeline to construct coherent and temporally consistent long-horizon trajectories from user-app sessions. It provides complementary text and multimodal settings covering multi-hop and temporal reasoning, knowledge updating, and implicit preference inference. Specifically, MobileMem enables agents to remember the past, understand the present, and adapt to the future. By modeling experiences rather than isolated facts, MobileMem moves memory beyond information retrieval toward experiential intelligence for continuous personal learning.
Sam: So the health app knows your prescriptions, the calendar knows your appointments, but they all contribute to that shared layer?
Alex: Precisely. That shared layer acts like a coordination point — a place where different knowledge sources can be cross-referenced without any single app needing to know everything. And critically, it all stays on the device. Nothing has to leave.
Sam: How does the system actually build that shared layer from raw, messy data?
Alex: That's where the synthesis engine comes in. Raw data — a photo, a voice memo, a search query — gets processed and compressed into what the paper calls "interaction trajectories." Imagine taking a week of your life and distilling it into a short, structured summary: where you went, what you were thinking about, what decisions you made. The engine does that automatically, turning fragmented inputs into coherent personal history.
Sam: That compression step seems important. You can't store every raw photo forever, but you can store the meaning behind it.
Alex: Exactly. And that distinction — storing meaning rather than raw data — is what makes long-term memory feasible on a device with limited storage. The system keeps what matters and lets go of the rest.
Sam: The paper also evaluates how well this actually works. How do they test a system like this?
Alex: That's one of the more thoughtful parts of the research. Rather than testing the assistant on simple, artificial prompts, they evaluate it on realistic, extended personal histories — the kind of messy, evolving data that reflects how people actually live. The argument is that a system can look good on simple tests but fall apart when it needs to handle months of overlapping, sometimes contradictory information.
Sam: And the results suggest that evaluating on authentic data makes a meaningful difference in how well the system handles complex personal histories?
Alex: That's what the paper reports, yes. Systems trained and tested on realistic trajectories handle nuanced, long-horizon questions more reliably than those optimized for simpler benchmarks.
Sam: Though the authors are upfront about the limitations here.
Alex: They are. Synthetic data used in training can carry residual noise, and modeling how human psychology actually shifts over time is genuinely difficult. People change — their preferences, their priorities, their relationships — in ways that are hard to capture in a static profile. The paper acknowledges that future versions will need more dynamic models to keep pace with that kind of drift.
Sam: So this is a meaningful step forward, but not a finished solution.
Alex: That's a fair characterization. The authors also gesture toward where this could go next — pulling in continuous data streams from wearables or augmented reality glasses, building something closer to a unified, privacy-preserving lifelong memory that spans all your personal devices.
Sam: A system that knows your history the way a long-term doctor or close family member might — but without the privacy risks of handing all of that to a corporation.
Alex: That's the aspiration the paper points toward. Whether the engineering can fully deliver on it remains an open question, but the architecture described here is a concrete step in that direction. Thanks for listening to ResearchPod.