Existing research on LLM agent security mainly focuses on prompt injection and unsafe input/output behaviors. However, as agents increasingly rely on third-party tools and MCP servers, a new class of supply-chain threats has emerged, where malicious behaviors are embedded in seemingly benign tools, silently hijacking agent execution, leaking sensitive data, or triggering unauthorized actions. Despite their growing impact, there is currently no comprehensive benchmark for evaluating such threats. To bridge this gap, we introduce SC-Inject-Bench, a large-scale benchmark comprising over 10,000 malicious MCP tools grounded in a taxonomy of 25+ attack types derived from MITRE ATT&CK targeting supply-chain threats. We observe that existing MCP scanners and semantic guardrails perform poorly on this benchmark. Motivated by this finding, we propose ShieldNet, a network-level guardrail framework that detects supply-chain poisoning by observing real network interactions rather than surface-level tool traces. ShieldNet integrates a man-in-the-middle (MITM) proxy and an event extractor to identify critical network behaviors, which are then processed by a lightweight classifier for attack detection. Extensive experiments show that ShieldNet achieves strong detection performance (up to 0.995 F-1 with only 0.8% false positives) while introducing little runtime overhead, substantially outperforming existing MCP scanners and LLM-based guardrails.
Alex: Welcome to another episode of ResearchPod. Sam, what are we looking at today?
Sam: This paper examines security risks in AI agents that use external tools from third-party servers. The key puzzle is that these agents can get hijacked by malicious code hidden inside tools that look harmless on the surface.
Alex: So agents pull in these tools dynamically, like calling an app for weather or time, but attackers slip bad code into them without changing how they seem to work?
Sam: Exactly. The tools keep their normal descriptions and give back correct answers, so defenses that just check words or inputs miss the hidden actions—like secretly sending data out over the network. The paper calls this a supply-chain attack, where the danger comes from tainted tool code the agent trusts blindly. They built a benchmark called SC-Inject-Bench with over 10,000 such malicious tools to test defenses.
Alex: Right, and current checks focus on the tool's words or outputs, not what it does behind the scenes on the network?
Sam: Yes. Existing MCP scanners—tools that verify how agents find and call these external functions—only inspect interfaces, like names and input formats, not the code itself. Semantic guardrails judge based on chat logs or descriptions, assuming bad intent shows up there. But these stealthy attacks preserve clean interfaces and outputs while doing things like data leaks or network floods during execution, invisible to those methods.
Alex: That sounds like a real blind spot. Like trusting a delivery service based on their menu, but not watching what they load in the van?
Sam: Precisely. The paper shows a practical example: a 'get_weather' tool that returns the right temperature but sneaks out your API keys through odd network queries. To catch this, they propose ShieldNet, which watches actual network traffic during tool use instead of just words.
Alex: Huh. So it shifts from checking what the tool *says* to what it *does* online. How does it actually grab that traffic without the tool noticing?
Sam: They set up a local setup that sits right in the middle of the connection between the AI agent and the outside internet. It copies every message going out or coming in, like a silent observer jotting down a phone conversation without the callers knowing. This is called a man-in-the-middle proxy, or MITM proxy for short. It also handles the scrambled parts of web traffic by unscrambling them on the fly.
Alex: Right—like decrypting secret messages in a spy novel. And it catches things even if the tool tries fancy tricks to hide?
Sam: Yes. Modern apps sometimes skip the usual path using quick connections called QUIC, but they block those during tests to force everything through the proxy. They also grab raw copies of all network packets—those are the tiny data bundles flying around—like recording every letter sent in a mail system. From there, the system turns those into a neat timeline of actions, such as a request for a web address or a burst of unusual messages.
Alex: So it spots weird patterns, like extra lookups or floods of junk data, that don't match a normal weather check?
Sam: Exactly. A poisoned tool might send your secret keys hidden in odd address lookups or flood another server with useless packets, while still giving the right temperature back. These network oddities stand out in the timeline because safe tools stick to simple, expected patterns—like a quick chat instead of sneaking off to call strangers. They feed this cleaned-up timeline to a small, trained model that spots the danger.
Alex: Huh. And does this actually work well without flagging normal stuff by mistake?
Sam: The paper shows it catches nearly all the bad cases while only wrongly tagging about 0.8 percent of safe ones. That's a clear improvement over methods that guess from words alone, because it directly sees the hidden actions. They tested it on new attacks and servers it hadn't seen, keeping the mistakes low even in real use.
Alex: Okay, how do they turn that raw timeline of network actions into something the detection system can actually judge?
Sam: They shape the timeline into a simple text list, where each action gets its own line. It includes the exact time, what kind of action it was—like looking up a web address or sending a web request—the computers involved, the connection type, and a short note on what happened. This creates a lightweight, step-by-step story of the network behavior that stays in the original order. They call this process serializing the event sequence.
Alex: So it's like turning a video of footsteps into a written log: time, direction, speed. And then what decides if it's safe or not?
Sam: That log goes straight to a detection model, which sorts it as either safe or one specific type of threat. As a starting point, they tried big language models prompted with the full log, which reason well but take too long and miss network details without special training. Instead, they fine-tuned a small, efficient model—Qwen3-0.6B—on examples from their data using supervised fine-tuning. This teaches it network patterns directly, cutting processing time while keeping accuracy high.
Alex: Huh, makes sense for speed in real chats. Does it handle ongoing tool use, like as the agent keeps calling tools back and forth?
Sam: Yes, for live sessions where network actions build up over time, they use streaming detection. New actions get added to the growing log in real time, and every so often—say, after a batch of them—they check just the latest chunk using a sliding window. If that chunk flags as risky, it alerts right away, catching multi-step threats without waiting for everything to finish.
Alex: So the data for training all this—how did they make sure it's realistic, pulling from real tools?
Sam: They built a benchmark by starting with real MCP servers from official lists and communities—109 of them with 984 tools. Using a refined set of 29 attack types from the MITRE ATT&CK framework—ones that show clear network signs like odd connections or data flows—they injected malicious scripts directly into the tool code. An automated pipeline found safe servers, wove in the bad scripts at key spots, had agents run the tools, and checked success through captured network traces in controlled setups. This gave nearly 20,000 verified runs, far more realistic than past benchmarks that skipped code changes or network proof.
Alex: So they wove real malicious behaviors into actual tool code from real servers, using specific attack patterns, and only kept the runs where everything fired correctly—like confirming the poison actually worked by spotting a telltale signal in the traffic.
Sam: Yes. To embed the attacks, a code-handling AI scanned each tool's files, picked safe spots to insert scripts that match attack recipes from MITRE ATT&CK—things like sneaky data sends or odd connections. These scripts include a simple test signal, like a quick ping, that shows up in network logs only if the attack runs. For checks, they ran agents in a controlled setup using natural questions that fit each tool, confirming both that the tool got called and the signal appeared for bad ones.
Alex: Right—like planting a tracer bullet to prove the gun fired. And with that solid data, how did ShieldNet stack up against other defenses?
Sam: They trained on about 6,000 examples, holding back some servers and three attack types to test on new stuff. Compared to static checkers that scan tool lists upfront—like Cisco AI or Ramparts—or ones that just watch chat inputs and outputs, ShieldNet caught nearly everything with few false alarms on safe tools. Traditional network watchers either missed most attacks or flagged too much normal traffic, and even big language models prompted on the logs were slower and less precise. The paper points to network views as key, since word-based methods miss the hidden network tricks.
Alex: Huh. So the fine-tuned small model beats bigger ones on speed and accuracy because it learned the exact patterns?
Sam: Exactly. It stayed strong on servers and attacks never seen in training because those patterns transfer across tools. Big models guessed okay but added delays unsuitable for live use, while ShieldNet added less overhead with better balance. Breaking down the traffic into those timed event lists, plus unscrambling secure connections, proved essential for reliable spotting.
Alex: Okay, so decrypting and structuring the events lets it see the full picture without slowing things too much. That addresses the blind spot in agent security nicely. But what doesn't it catch—any gaps left open?
Sam: It targets network-visible threats, so purely local actions—like deleting files or tweaking permissions without online signs—slip by. The paper suggests pairing it later with host monitoring or chat checks for fuller coverage. Still, for data leaks and remote tricks, network views fill a clear gap others ignore.
Alex: Makes sense—not a full shield, but strong where words-based methods fail. The takeaway seems to be shifting focus to what tools do online during runs.
Sam: Precisely. This work highlights network traces as a reliable guard for agent tool risks, with solid evidence on generalization and low overhead. It points toward safer ecosystems for third-party tools by blocking hidden threats early.
Alex: Yeah, a grounded step forward without overpromising. Thanks for joining us on ResearchPod.