PROEVENT: An Event-centric Benchmark for Proactive Agents

As large language models (LLMs) transition from reactive assistants—which wait for explicit user commands—to proactive agents, they must be able to autonomously track and manage user events. The authors investigate whether current LLMs can effectively maintain a user's timetable by extracting information from ongoing, noisy, and concurrent instant messaging conversations.

The authors developed ProEvent, a benchmark designed to evaluate proactive agents on three core tasks: response timing (deciding when to act), single-step correctness (performing the right operation), and multi-step correctness (maintaining a consistent schedule over time). The dataset consists of synthesized, realistic chat logs that incorporate complex real-world factors, such as dynamic negotiation, concurrent chat threads, and irrelevant noise. The researchers evaluated eight LLMs and agent pipelines, using metrics like False Detection Rate (FDR) and Timetable Success Rate (TSR) to measure performance.

The study reveals that current LLMs are far from achieving reliable proactive assistance. A significant finding is the systemic bias toward over-responsiveness: most models exhibit a high False Detection Rate, meaning they frequently trigger actions when no intervention is actually required. Furthermore, models struggle significantly with event cancellations and maintaining long-term consistency in a user's schedule. Even top-tier models like GPT-5.1 achieved a correct response rate of only 26.7% in the tested scenarios. Qualitative analysis suggests that models often fail to detect implicit events and struggle to reason from the user's first-person perspective.

Proactive agents have the potential to significantly reduce human cognitive load by automating scheduling and reminders. However, this study demonstrates that current models are prone to "overacting," which could lead to cluttered calendars and user frustration. By providing a standardized, event-centric benchmark, this work highlights the specific reasoning gaps—such as handling implicit information and noise—that developers must address to build truly dependable autonomous assistants.