Mingxuan Zhang, Jiahui Han, Dadi Guo, Songze Li, Guanchu Wang, Na Zou, Dongrui Liu, Xia Hu
5 min
Abstract
LLM-based agents are rapidly advancing, autonomously invoking external tools to complete multi-step tasks for users. However, agents often acquire more sensitive information than the task requires. Existing privacy benchmarks audit what the agent's response or outgoing actions disclose, but overlook the acquisition stage where data first enters the agent's context. The over-acquired information is then one careless action or one attack away from an outright leak. To assess its prevalence, we introduce \emph{PrivacyPeek}, a benchmark for evaluating acquisition-stage privacy leakage of LLM-based agents, with $1{,}182$ cases across $7$ acquisition behaviours and $16$ application domains. Specifically, \emph{Acquisition Inspection} examines the agent's tool-call trajectory, both the tools it invokes and the data it receives, to detect when it acquires sensitive information beyond the task scope. \emph{Probe Elicitation} then issues a follow-up probe and measures how readily an attacker could elicit sensitive information the agent acquired but did not disclose. Our experiments on $10$ LLM-based agents across $4$ model families show that the unnecessary acquisition of sensitive information is widespread. In addition, we observe a correlation between the task-completion capability and acquisition-stage leakage. Prompt-level defences reduce only a small fraction of acquisition-stage leakage, leaving the majority unmitigated. These results make auditing acquisition-stage privacy both urgent and necessary. Our dataset and code are available at https://github.com/Xuan269/PrivacyPeek-Resource.
Alex: That's a significant finding. It suggests that simply making these tools more powerful doesn't automatically make them safer—it may actually do the opposite.
Sam: That's the implication. And the paper makes another point worth noting: telling the AI to "be careful" doesn't fix this. Simple instructions added to the prompt—the text you use to direct the AI—don't reliably stop the over-collection. The researchers argue the problem isn't one of attitude; it's one of architecture. It's baked into how these systems are built.
Alex: Can you give a concrete example of what that looks like in practice?
Sam: Sure. They track a behavior they call "Excess-Field Access." Imagine an agent is asked to check whether a patient's blood pressure result is above a certain threshold. To do that, it only needs that one number. But instead, the agent pulls the entire medical record—name, address, every test result on file. It got the job done, but it handled far more sensitive information than the task required.
Alex: So it's not that the agent did anything wrong in terms of the result. It just wasn't careful about what it picked up along the way.
Sam: Precisely. And that pattern is consistent. In their tests, the rate at which agents grabbed data outside the minimum necessary scope exceeded fifty percent on some models. More tellingly, the agents that performed best on the actual task were often the ones with the highest rates of unnecessary data collection. Success at the job and restraint with data weren't correlated—they were inversely related.
Alex: So if standard prompting doesn't fix this, what does the paper suggest actually would?
Sam: The paper points toward what they call "context-aware memory boundaries." The idea is that instead of asking an AI to choose to be careful, you build the system so it is structurally incapable of loading data outside a defined scope. Think of it like a security clearance system: you're not relying on an employee's judgment about what they should or shouldn't read—you physically restrict which files they can open in the first place. The agent wouldn't be making a decision to ignore irrelevant data; it simply wouldn't be able to see it.
Alex: That's a meaningful distinction. It's the difference between a rule posted on the wall and a locked door.
Sam: And that's really the paper's core argument. Right now, we're mostly relying on the posted rule. The research suggests we need the locked door—privacy protections that are structural, not advisory.
Alex: It's a useful frame for thinking about where AI development needs to go. Building in restraint from the start, rather than patching it in afterward. Thanks for walking us through this one. And thanks to everyone listening to ResearchPod.