Chung-En Sun, Linbo Liu, Tsui-Wei Weng
5 min
Abstract
Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks -- a phenomenon we term the cold-start safety gap. To study this systematically, we introduce Safety Over Depth for Agents (SODA), a benchmark that controls how many regular agentic tasks the agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms that model hidden states gradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regular agentic tasks themselves are the primary driver of safety, while the agent's own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-source safety benchmarks (AgentHarm, Agent Safety Bench) and utility benchmarks (BFCL, API-Bank), confirming that warming up the agent with regular agentic tasks before deployment makes it safer and preserves full capability. Based on these findings, we recommend a simple deployment strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap
Alex: So the warm-up isn't about the AI learning from its own past work. It's about being shown the type of environment it's operating in.
Sam: Precisely. When the model sees these routine requests, it seems to recognize that it's in an agentic role—one where it has real tools and real consequences—and that recognition triggers its safety alignment. It's like a worker putting on a uniform. The act of getting into character matters more than the specific tasks completed.
Alex: Did they check whether this holds up beyond their own test environment?
Sam: They did. They tested it on two external safety benchmarks, and the effect held. Agents that completed five to ten routine tasks before encountering a harmful request were significantly better at refusing it. The pattern was consistent.
Alex: But here's the practical concern—does making an agent do extra warm-up tasks slow it down or make it worse at its actual job?
Sam: That depends on how the warm-up is done. If the agent performs real, meaningful tasks, its ability to handle complex problems stays strong. But if you try to shortcut it by feeding the agent fake or scripted responses to simulate a warm-up, the agent can become less capable—it hasn't actually engaged with its environment, so the useful side of its performance suffers even as the safety improves.
Alex: So the quality of the warm-up matters, not just the quantity.
Sam: Right. And that points to a genuine trade-off. Waiting for five to ten real tasks introduces a delay at the start of every session. In fast-moving environments where users expect an immediate response, that friction is a real design consideration.
Alex: Is there a way to get the safety benefit without the delay?
Sam: That's where the research points next. One possibility is what you might call a pre-flight calibration—a short, invisible sequence built into the model that simulates the warm-up effect without requiring actual tasks. It would give the agent the context it needs to behave safely from the first moment, without making the user wait. That's still a future direction, though, not a current solution.
Alex: What about the approach of pre-filling the conversation with examples of the agent refusing harmful requests? That sounds like it could work.
Sam: Researchers have explored that. The problem is it tends to make the model over-cautious—it starts refusing perfectly normal, helpful requests because it's been primed to be suspicious. It also degrades the model's ability to use its tools correctly. So you end up with an agent that's harder to misuse but also harder to use at all. The natural warm-up approach avoids that penalty.
Alex: So the cleaner solution is just letting the agent do its actual job for a few minutes before it handles anything sensitive. Simple, but it requires a real shift in how we think about safety.
Sam: That's the core takeaway. We tend to treat AI safety as a fixed property—something baked into the model once and then stable forever. This research suggests it's actually a dynamic state that depends on the context of the interaction. By managing that context thoughtfully, we can make these systems meaningfully more reliable without rewriting a single line of their underlying code.
Alex: It's a useful reminder that even the most carefully designed systems can behave differently depending on when and how you engage with them. Thanks for listening to ResearchPod.