Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation

As LLMs become ubiquitous, organizations require safety guardrails that are both effective and efficient. Current industry standards rely on decoder-based LLM judges, which are often slow and expensive to deploy at scale. This paper investigates whether modern, fine-tuned encoder classifiers (specifically from the ModernBERT family) can serve as high-performance, low-latency alternatives for identifying harmful LLM outputs without sacrificing accuracy.

The researchers conducted a systematic comparison using a massive dataset of over 560,000 (prompt, response) pairs generated across 16 different LLMs. They employed a panel of seven diverse safety judges—including proprietary LLMs, fine-tuned safety models, and rubric-based scorers—to generate training labels. To address the inherent disagreement among these judges, the authors implemented a high-confidence filtering strategy, retaining only unanimous or strong-majority consensus labels. The resulting encoder models were then benchmarked against frontier LLM judges on a held-out, out-of-distribution dataset consisting of 42,899 examples from JailbreakBench and AILuminate.

The study demonstrates that encoder-based classifiers are highly effective at detecting harmful content, often matching the performance of much larger decoder-based models. Because encoders process the entire input bidirectionally in a single pass, they are particularly well-suited for classification tasks where understanding the full context is critical. The authors provide a detailed breakdown of performance across different attack techniques—including single-turn prompting, decomposition, escalation, and context manipulation—showing that encoders remain robust even when faced with complex, multi-step adversarial strategies.

This research provides a practical blueprint for developers and security teams looking to optimize their LLM safety pipelines. By demonstrating that encoders can replace expensive LLM-as-a-judge setups for many classification tasks, the paper offers a path toward reducing the latency and financial overhead of production-grade AI safety guardrails without compromising on the ability to detect adversarial threats.