Zongrui Wang, Xiangyang Zhu, Sicheng Wang, Han Wang, Dingyi Rong, Zeyu Zhang, Chunyi Li, Yue Shi, Kaiwei Zhang, Zicheng Zhang, Yuan Tian, Qi Jia, Yan Teng, Wei Sun, Ning Liu, Guangtao Zhai
5 min
Abstract
Multimodal safety moderation requires distinguishing risks arising from visual content, user intent, and assistant behavior. Existing safeguards, however, are typically trained for a single judgment target and reduce safety assessment to a binary decision. Consequently, risk becomes difficult to compare across a multimodal interaction, and ambiguous cases are obscured. We introduce SafeAtlas-VL, a dataset of 1.5M training instances that places image-, request-, and response-level judgments on a five-level ordered scale. We curate a broad collection of safety-relevant data from both real-world and synthetic sources and apply a disagreement-aware annotation procedure. The resulting dataset spans 15 harm categories and 55 fine-grained subcategories, covering a broad range of multimodal safety scenarios. We also construct SafeAtlas-Bench, a held-out set of 5,000 instances for evaluating five-level predictions and continuous risk scores. Upon this dataset, we train the SafeAtlas Guard series of models via target-conditioned tuning for multimodal safety detection. Our models not only perform five-way classification of safety levels but also map safety to continuous scores through a soft cumulative ordinal head. Experimental results demonstrate that guard models trained on our dataset exhibit strong generalization: even without using the training sets of other benchmarks, they achieve competitive performance on the corresponding test sets. Notably, our 8B model attains the overall best performance, outperforming the previous SOTA by approximately 4% in F1 score. Code, data, and models are released to support further research. Warning: this paper contains example data that may be offensive, harmful, graphic, or disturbing.
Alex: So the architecture is encoding the assumption that risk is a gradient, not a set of discrete bins.
Sam: That's the right way to read it. And the payoff is that the system produces continuous risk scores rather than hard labels, which enables graduated interventions downstream. An application can set its own threshold based on context — a sensitive query in an educational setting might clear a lower bar and receive a nuanced response, while the same query in a different deployment context triggers a refusal.
Alex: Which is a qualitatively different relationship between the model and the application layer than a binary gate.
Sam: It is. On the performance side, their 8B model achieves state-of-the-art results across several multimodal safety benchmarks, with roughly a four-point F1 improvement over the previous best. That's the load-bearing number — it's not a marginal gain, and it holds across the multimodal inputs that make this problem hard, where you have to reason jointly about image and text.
Alex: Where does a careful referee push back?
Sam: Two places. The first is dual-use risk. The dataset contains harmful content by construction — that's unavoidable if you want to train a model that understands the boundary. But a sufficiently motivated actor could use a well-calibrated guard model to probe exactly where the refusal threshold sits, which is essentially a map for optimizing adversarial inputs. The authors acknowledge this, but it's a structural tension in the design, not something you can fully engineer away.
Alex: And the second?
Sam: The taxonomy itself. Safety definitions are culturally and policy-dependent, and what this model labels "unsafe leaning" reflects the specific judge-model consensus baked into training — not an objective ground truth. The five-level scale is a useful abstraction, but it's one particular operationalization. Deploy this in a different regulatory context or with a different policy alignment, and the label assignments may not transfer cleanly. That's a distribution shift problem the paper doesn't fully address.
Alex: So the system is a snapshot of a specific policy alignment, and the ordinal structure doesn't make it more universal — it just makes it more precise within that alignment.
Sam: That's the right framing. The contribution is real — moving from binary to ordinal is a genuine architectural improvement, and the disagreement-aware annotation is a methodologically sound way to construct the middle categories. But the value of those continuous risk scores depends entirely on whether the underlying taxonomy matches the deployment context. That's the constraint that most limits how far this result generalizes.
Alex: That's a useful place to land. Thanks for the walkthrough, Sam.
Sam: Happy to. Thanks for listening to ResearchPod.