Sitong Gong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, Yifei Huang
5 min
Traditional intelligent assistants are reactive, waiting for explicit user queries before acting. This paper addresses the challenge of building a proactive assistant that can observe continuous egocentric (first-person) video and autonomously decide when to offer help, without being prompted. The authors aim to move beyond simple event-triggered responses by enabling agents to reason over long-term user history, habits, and current activities.
To formalize this, the authors propose a new paradigm for proactive assistance, reframing it as a context-dependent decision problem. They introduce two primary contributions:
EgoServe: A large-scale benchmark for proactive assistance. It organizes 3,000+ service instances into four temporal horizons—ranging from immediate safety alerts to long-term habit coaching—across 10 distinct service categories. This allows for a structured evaluation of how well an agent understands the timing and necessity of an intervention.
EgoMemo: A training-free, memory-augmented agent. EgoMemo maintains three types of memory: multi-scale temporal summaries (hierarchical organization of video captions), a semantic knowledge graph (tracking entity relationships), and visual embedding archives (for similarity-based retrieval). At each timestep, the agent performs retrieval-augmented reasoning to decide whether to intervene and what to say, ensuring responses are grounded in both immediate and historical context.
Proactive assistance is a critical step toward truly helpful wearable AI. By moving from reactive to proactive, assistants can provide value in scenarios where a user might be too busy or distracted to ask for help, such as correcting a procedural error in a cooking task or reminding a user of a forgotten habit. The EgoServe benchmark provides a standardized way to measure this capability, while EgoMemo demonstrates that complex, long-horizon reasoning can be achieved without expensive retraining of large models.
When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at \href{https://sitonggong.github.io/EgoServe-page/}{Vinci2}.
Alex: How does it know which past moment is actually relevant right now? If I'm cooking dinner, how does it connect that to something I did three days ago?
Sam: It uses what the paper calls streaming retrieval—an indexing process that runs in real-time. Think of it like a library that catalogs every book the moment it arrives on the shelf. The system builds three separate pathways for finding relevant memories: one based on time, one based on meaning, and one based on what things look like visually. When the agent needs to decide whether to help, it searches all three simultaneously and uses a vision-language model—an AI that understands both images and text—to weave those findings into a coherent picture of what's happening.
Alex: That sounds computationally heavy. Does it get bogged down?
Sam: The authors designed it to avoid that by only reconstructing specific, relevant segments rather than processing the entire history. It pulls the right notes from its index rather than re-reading the whole book, so to speak.
Alex: Where does the data come from in the first place? How do they build the dataset that teaches the system what good assistance looks like?
Sam: They use a semi-automated pipeline. A powerful AI model watches video and turns it into structured descriptions of what's happening. Then human experts review those descriptions to decide whether a potential suggestion is genuinely useful or just noise. The humans are essentially the quality filter—they make the judgment calls the model can't yet make reliably on its own.
Alex: And how do they measure whether the AI is actually being helpful rather than just annoying?
Sam: They use an automated scoring system that evaluates two things. First, rationality—does the advice make logical sense given the situation? Second, effectiveness—is the advice actually actionable, something you could do right now? The study found that effectiveness scores tended to be higher than rationality scores, which suggests the system is better at giving you something useful to do than at explaining why it's suggesting it.
Alex: That's an interesting gap. It can tell you what to do before it can fully articulate why.
Sam: Which is actually not unlike how human intuition works sometimes. The deeper shift here is moving AI from "can I answer this question?" to "should I say something right now?"—and that changes the entire dynamic of how we interact with these systems. Whether that's a welcome change or an intrusive one probably depends a great deal on how well the system learns to read the room.
Alex: It raises a real question about where the line is between a helpful nudge and an unwanted interruption.
Sam: And that's likely where a lot of the future work on this will focus. Getting the timing right is at least as important as getting the content right. A perfect suggestion delivered at the wrong moment is still an interruption.
Alex: A useful reminder that the hardest part of being helpful—for humans or machines—is knowing when to speak and when to stay quiet. Thanks for listening to ResearchPod.