Prompt injection poses significant security risks to LLM agents. Efficient and effective red-teaming is therefore critical, both for evaluating these risks and for collecting training data to improve defenses. Existing state-of-the-art prompt injection red-teaming methods primarily rely on reinforcement learning (RL), producing attacker models that often generalize poorly to new target LLMs. In this work, we develop PIMiner, an agentic system for prompt injection red-teaming. During training, PIMiner is trained on a sequence of (dataset, target model) pairs and builds a strategy library from scratch. At test time, the learned strategy library can be directly transferred to a previously unseen target LLM without additional training. PIMiner requires only a small number of queries to a target agent (e.g., 10) per test sample. Experimental results demonstrate that PIMiner achieves strong performance. On IPIArena, it attains a 76.2% ASR against Gemini-2.5-Pro, 61.9% ASR against GPT-5.1, and 42.9% ASR against Claude-Sonnet-4.5. On AgentDojo, it achieves an 86.7% ASR against Gemini-2.5-Pro, 53.3% ASR against GPT-5.1, and 40.0% ASR against Claude-Sonnet-4.5.
Alex: Welcome to another episode of ResearchPod. Today, we're discussing a new framework called PIMiner, designed to automate security testing for AI agents.
Sam: So this paper is basically asking: why do we keep starting from scratch every time we test an AI for security flaws?
Alex: Exactly. The central claim is that we don't need to retrain models to find vulnerabilities. We just need to remember how we broke them last time.
Sam: And the core problem with current methods is that they're either too expensive, or they don't learn from their mistakes?
Alex: Right. The dominant approach right now is something called reinforcement learning. Think of it like training a dog with treats—the AI "attacker" plays thousands of rounds against the target, slowly learning what works through repetition. That takes a huge amount of computing power and data.
Sam: So instead of training a new attacker from scratch for every target, this system builds a library of attack strategies it can reuse?
Alex: That's the core idea. Think of it like a cookbook for security testing—one that gets smarter with every test run. The researchers call this "red-teaming," which is just the practice of deliberately trying to break a system to find its weak points before a real attacker does.
Sam: But how does it decide which recipe to use when it encounters a brand-new AI it hasn't seen before?
Alex: It uses a component called a "strategy router." Think of it like a librarian who looks at the new target and pulls only the most relevant files from the shelf—rather than loading the entire library at once, which would be slow and expensive.
Sam: So it stays fast and affordable even as the library grows?
Alex: Correct. And if no strategy fits at all—if the target is genuinely unlike anything the system has seen—there's a fallback. It treats the problem as brand new, builds a strategy from the ground up, and adds it to the library for next time.
Sam: So it's a hybrid. It uses past knowledge when it can, but it's not afraid to improvise when it's truly stumped.
Alex: Precisely. And what happens after each attempt—whether it succeeds or fails—is where things get interesting. The system has what the paper calls an "experience digester." It records what happened, analyzes why an attack worked or didn't, and updates the library accordingly.
Sam: So every test run becomes a lesson for the next one. That's a meaningful shift from brute force.
Alex: It is. And it connects to something the paper calls "grey-box" versus "black-box" access. In a black-box test, the system only sees the AI's final response—like watching someone answer a question without knowing their thought process. In a grey-box test, it can see the intermediate steps the AI took to get there.
Sam: So grey-box is like having a transcript of the suspect's thoughts, while black-box is just their final statement?
Alex: That's a good way to put it. Those intermediate steps make it much easier to diagnose exactly where an attack broke down—and that diagnosis feeds directly back into the library.
Sam: It's really about building a collective memory for the security community. Instead of every engineer hitting the same wall, they just read the failure notes from the last person who tried.
Alex: That's the core of it. And it has a practical cost advantage too. Traditional reinforcement learning requires enormous compute budgets to train an attacker model. Because PIMiner treats testing as a search problem—not a training problem—the costs stay manageable. The paper also notes it uses something called prompt caching, which means it stores the parts of the strategy library that don't change between tests, so it isn't paying to re-process the same information over and over.
Sam: So the initial investment is in building the library, but every new test makes future tests cheaper?
Alex: Exactly. You're not just paying for one test—you're building a reusable asset. And because the library is transferable across different models, the same strategies that worked against one AI can be applied, with adjustments, to the next one.
Sam: So the key insight is really about transferability. The knowledge isn't locked to one specific model—it travels.
Alex: That's the main takeaway. The paper reports meaningful success rates against major commercial models, achieved with relatively few queries per test. The research suggests that by treating attack history as a structured, reusable asset, security testing becomes far more practical—without requiring massive compute budgets.
Sam: It's a shift from asking "how do we build a smarter attacker?" to "how do we make sure we never forget what we've already learned?"
Alex: That's a precise way to frame it. And it points toward something important for AI safety more broadly—that systematic, memory-based approaches may prove more durable than methods that start from scratch each time. Thanks for listening to ResearchPod.