Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos | Sitong Gong et al. | ResearchPod

Transcript: Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Alex: Welcome to another episode of ResearchPod. Today, we're looking at how AI assistants might move from waiting for commands to helping before we even ask.

Sam: We're discussing a system called Vinci2. The core puzzle is simple: most AI is reactive. You ask, it answers. This paper argues a truly helpful assistant should be proactive—using the context of your life to offer support before you even know you need it.

Alex: So it's about moving from a passive tool to a partner that actually understands what we're doing?

Sam: Exactly. Current systems treat every event as a prompt. This research treats assistance as a decision-making task. The agent looks at your history and habits to decide whether it should even interrupt you in the first place.

Alex: That's a meaningful distinction. It's like the difference between a coworker who only answers questions and one who notices you're struggling and quietly offers a tip without being asked.

Sam: That's a good way to put it. To test this idea, the authors created something called EgoServe—a collection of video data covering different time scales, from immediate safety warnings to long-term habit coaching. The key question is whether AI can remember things across days, not just seconds.

Alex: If it has to remember things from days ago, isn't that overwhelming? How does it track everything without drowning in data?

Sam: That's exactly the challenge. The system uses a component called EgoMemo, which works a bit like how human memory is organized. Think of it in three layers. There's working memory—what's happening right now. There's episodic memory—things that happened earlier today or yesterday. And there's semantic memory—your longer-term habits and patterns. Each layer handles a different time horizon.

Alex: So instead of processing every single frame of video, it's building summaries at different levels?

Sam: Precisely. It compresses video into clips and sessions. When the AI needs to make a decision, it retrieves only the relevant pieces—like a student looking up a specific note in a textbook rather than re-reading the whole thing. That selectivity is what makes it practical.

Alex: So it's building a kind of searchable map of your life, and it pulls up the right memory when the current moment calls for it.

Sam: That's it. And because it organizes data hierarchically, the agent can decide whether to speak up based on whether the current moment matches a known pattern. It's also designed to be training-free, meaning it doesn't need to be taught every possible scenario from scratch. It reasons from what it already knows.

Alex: How does it know which past moment is actually relevant right now? If I'm cooking dinner, how does it connect that to something I did three days ago?

Sam: It uses what the paper calls streaming retrieval—an indexing process that runs in real-time. Think of it like a library that catalogs every book the moment it arrives on the shelf. The system builds three separate pathways for finding relevant memories: one based on time, one based on meaning, and one based on what things look like visually. When the agent needs to decide whether to help, it searches all three simultaneously and uses a vision-language model—an AI that understands both images and text—to weave those findings into a coherent picture of what's happening.

Alex: That sounds computationally heavy. Does it get bogged down?

Sam: The authors designed it to avoid that by only reconstructing specific, relevant segments rather than processing the entire history. It pulls the right notes from its index rather than re-reading the whole book, so to speak.

Alex: Where does the data come from in the first place? How do they build the dataset that teaches the system what good assistance looks like?

Sam: They use a semi-automated pipeline. A powerful AI model watches video and turns it into structured descriptions of what's happening. Then human experts review those descriptions to decide whether a potential suggestion is genuinely useful or just noise. The humans are essentially the quality filter—they make the judgment calls the model can't yet make reliably on its own.

Alex: And how do they measure whether the AI is actually being helpful rather than just annoying?

Sam: They use an automated scoring system that evaluates two things. First, rationality—does the advice make logical sense given the situation? Second, effectiveness—is the advice actually actionable, something you could do right now? The study found that effectiveness scores tended to be higher than rationality scores, which suggests the system is better at giving you something useful to do than at explaining why it's suggesting it.

Alex: That's an interesting gap. It can tell you what to do before it can fully articulate why.

Sam: Which is actually not unlike how human intuition works sometimes. The deeper shift here is moving AI from "can I answer this question?" to "should I say something right now?"—and that changes the entire dynamic of how we interact with these systems. Whether that's a welcome change or an intrusive one probably depends a great deal on how well the system learns to read the room.

Alex: It raises a real question about where the line is between a helpful nudge and an unwanted interruption.

Sam: And that's likely where a lot of the future work on this will focus. Getting the timing right is at least as important as getting the content right. A perfect suggestion delivered at the wrong moment is still an interruption.

Alex: A useful reminder that the hardest part of being helpful—for humans or machines—is knowing when to speak and when to stay quiet. Thanks for listening to ResearchPod.

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Sitong Gong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, Yifei Huang

Jul 13, 2026·4:59·arxiv·Original Paper

Proactive AssistanceEgocentric VideoEgoServeEgoMemoEgocentric Video UnderstandingRetrieval-Augmented Generation

Listen

5 min

Briefing

Abstract

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at \href{https://sitonggong.github.io/EgoServe-page/}{Vinci2}.

Research Question

Traditional intelligent assistants are reactive, waiting for explicit user queries before acting. This paper addresses the challenge of building a proactive assistant that can observe continuous egocentric (first-person) video and autonomously decide when to offer help, without being prompted. The authors aim to move beyond simple event-triggered responses by enabling agents to reason over long-term user history, habits, and current activities.

Approach

To formalize this, the authors propose a new paradigm for proactive assistance, reframing it as a context-dependent decision problem. They introduce two primary contributions:

EgoServe: A large-scale benchmark for proactive assistance. It organizes 3,000+ service instances into four temporal horizons—ranging from immediate safety alerts to long-term habit coaching—across 10 distinct service categories. This allows for a structured evaluation of how well an agent understands the timing and necessity of an intervention.
EgoMemo: A training-free, memory-augmented agent. EgoMemo maintains three types of memory: multi-scale temporal summaries (hierarchical organization of video captions), a semantic knowledge graph (tracking entity relationships), and visual embedding archives (for similarity-based retrieval). At each timestep, the agent performs retrieval-augmented reasoning to decide whether to intervene and what to say, ensuring responses are grounded in both immediate and historical context.

Why It Matters

Proactive assistance is a critical step toward truly helpful wearable AI. By moving from reactive to proactive, assistants can provide value in scenarios where a user might be too busy or distracted to ask for help, such as correcting a procedural error in a cooking task or reminding a user of a forgotten habit. The EgoServe benchmark provides a standardized way to measure this capability, while EgoMemo demonstrates that complex, long-horizon reasoning can be achieved without expensive retraining of large models.

Key Terms and Definitions

Egocentric Video — First-person video captured from a wearable device, providing a continuous stream of the user's perspective and environment.
Proactive Assistance — An agent-initiated interaction where the system autonomously decides to provide help based on observed context, rather than waiting for a user query.
Temporal Memory Horizon — The duration of historical context required to provide a specific service, ranging from immediate (seconds) to long-term (days).
Retrieval-Augmented Reasoning — A process where the agent queries its internal memory stores to gather relevant historical information before making a decision or generating a response.
EgoMemo — The proposed training-free agent architecture that uses hierarchical memory and knowledge graphs to support context-aware decision-making.

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos: Research Paper Podcast