SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE–ASK–REFUSE Routing

Standard LLM agent safeguards typically use a binary classification (safe vs. unsafe) to decide whether to allow an action. This approach is often too blunt: it either interrupts the agent for low-risk tasks (eroding autonomy) or fails to catch nuanced risks that require human judgment. The authors ask: can we reframe agent safety as a three-way routing problem—{EXECUTE, ASK, REFUSE}—to better balance autonomy, safety, and human oversight?

The authors introduce Safety Sentry, a lightweight guard model trained to classify agent actions into one of three categories. Unlike binary guards, Safety Sentry is trained on a custom corpus of 9,203 step-level records across nine real-world enterprise services (e.g., file sharing, ticketing, and email). The training data includes persona-conditioned memories, allowing the model to adapt its routing decisions based on user-specific risk preferences. A key technical innovation is the use of a single decoding-time threshold (τ), which allows developers to adjust the agent's autonomy-oversight balance post-deployment without needing to retrain the model.

Safety Sentry significantly outperforms both open-weight and frontier closed-source models in accuracy and safety-related recall. While baseline models often struggle with the ASK decision—frequently defaulting to either over-caution or over-autonomy—Safety Sentry maintains a sharp decision boundary. It successfully controls both directional error rates (Over-Ask Rate and Under-Ask Rate) simultaneously. Furthermore, the model demonstrates that a single fixed checkpoint can be tuned via the decoding threshold to serve different deployment environments, ranging from highly cautious to highly autonomous, without requiring additional training.

This work addresses a critical bottleneck in agentic systems: the "alarm fatigue" caused by excessive, poorly calibrated human-in-the-loop prompts. By providing a principled way to distinguish between actions that are clearly safe, clearly dangerous, and those that require human clarification, Safety Sentry enables more reliable and user-friendly agent deployment in sensitive enterprise environments.