arxiv

Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks

MD Azizul Hakim, Md Shihab Uddin, Talha Ibne AnichJul 1, 2026Original

ResearchPod Summary

Research Question and Approach

This study investigates whether lightweight machine learning models—designed for resource-constrained edge deployment in Industrial Internet of Things (IIoT) networks—can maintain their detection performance when applied to a different network without retraining. The authors evaluate four representative architectures (a decision tree and three compact neural networks) trained on the Edge-IIoTset dataset and tested on two independent, structurally distinct datasets: Gotham 2025 and WUSTL-IIoT-2021. To ensure a fair comparison, the researchers restricted the feature set to 16 common attributes available across all three sources, specifically using coarse port-category buckets instead of raw port numbers to avoid known shortcut learning.

Main Findings

The study reveals a consistent and substantial collapse in detection performance across all four architectures when applied to unseen networks. Under natural, imbalanced class distributions, F1 scores dropped from approximately 0.97 in-domain to between 0.09 and 0.28 in cross-domain scenarios. Explainability analysis using SHAP values demonstrated that these models rely heavily on coarse port-category features. Crucially, the most influential categories were found to occur at vastly different rates in the target domains compared to the source domain (up to 435 times more frequent in the training set), indicating that coarsening port resolution merely relocates the shortcut rather than removing it.

Furthermore, the authors show that using artificially balanced class distributions—a common practice in prior research—significantly overstates performance and can even reverse the perceived difficulty of different target networks. Finally, the ability to recover performance through limited fine-tuning on target-domain data was found to be highly architecture-dependent, with no single model consistently outperforming others across all metrics.

Why It Matters

These results challenge the prevailing assumption that high within-domain accuracy is a sufficient indicator of deployment readiness for IIoT security. By demonstrating that generalization failure is a systematic issue, the paper argues that cross-network evaluation under realistic, imbalanced class distributions must become a standard requirement for research in this field. It highlights that efficiency, adversarial robustness, and cross-domain generalization are independent axes of model performance, and that researchers must evaluate all three to provide a realistic assessment of a model's utility in critical infrastructure.

Paper Details

arXiv: 2607.00553
OpenAlex: W4386645116

Topics

Cross-Domain GeneralizationIIoT Intrusion DetectionShortcut LearningClass ImbalanceExplainability AnalysisCross-dataset evaluationLightweight Intrusion DetectionFeature relianceClass imbalanceFine-tuningCross-Dataset GeneralizationIntrusion Detection Systems (NIDS)

Podcast transcript

Alex: Welcome to another episode of ResearchPod. Today, we are examining a study on how lightweight machine learning models perform when tasked with spotting cyberattacks in industrial networks.

Sam: So this paper is basically asking whether these security models—designed to be small and fast enough to run on devices at the edge of a network—actually hold up when you move them from a controlled lab setting into the real world?

Alex: Exactly. And the answer the study gives is a cautious no. These models often report near-perfect accuracy during testing, but that performance frequently collapses the moment they encounter traffic from a different, unseen network.

Sam: That sounds like a serious problem. Is it just that the new networks look different, or are the models actually learning the wrong things from the start?

Alex: The researchers suggest it is the latter. They call it "shortcut learning." Imagine you are training a security guard to spot shoplifters. Instead of teaching them to watch for suspicious behavior—like concealing items or avoiding staff—you only ever show them footage from one store, where, by coincidence, every shoplifter happens to be wearing a red baseball cap. The guard learns: red cap equals threat.

Sam: And then you move that guard to a different store where nobody wears red caps, and suddenly they cannot identify anyone as a threat—even if someone is obviously stuffing items into a bag.

Alex: That is exactly what is happening here. The models latch onto specific patterns in the training data—things like which categories of network ports appear most often in attack traffic—and treat those patterns as the signal for danger. They are not learning what an attack actually looks like. They are learning what the training dataset looks like.

Sam: I have heard that researchers try to address this by grouping port numbers into broader categories, rather than using the specific numbers themselves. The idea being that if you make the data less specific, the model cannot just memorize individual ports.

Alex: That is the intention, yes. But this study found that the technique does not actually remove the shortcut. It just moves it. Instead of fixating on a specific port number, the model now fixates on the relative frequency of these broader port categories—and those frequencies are still highly specific to the training dataset, not to real-world attack behavior in general.