TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models

Large Reasoning Models (LRMs) have become increasingly powerful by generating detailed reasoning traces—intermediate steps that explain their thinking. While these traces improve performance, they also introduce new safety risks: a model might generate a safe final response while its reasoning trace contains dangerous or prohibited content. Current safety benchmarks and guardrail models focus almost exclusively on the final output or the user prompt, leaving these critical reasoning traces unexamined.

To address this gap, the authors introduce TRACE, an evidence-grounded benchmark designed to evaluate safety across the entire LRM inference pipeline: the prompt, the reasoning trace, and the final response. The dataset includes nearly 2,000 prompts spanning nine risk categories and ten attack strategies. For each prompt, four different LRMs generate reasoning traces and final responses. The researchers then employed a multi-stage annotation process using powerful LLMs and human experts to label the safety of each component and extract verifiable, verbatim evidence from the source text to justify those labels.

Evaluating 18 prominent guardrail models on TRACE yielded two primary insights:

1. Reasoning Traces are Harder to Moderate: Across the board, guardrail models performed worse when evaluating reasoning traces than when evaluating prompts or final responses. This suggests that the complex, often verbose nature of reasoning traces makes them a blind spot for current safety systems.
2. Evidence Attribution is Poor: Even the best-performing models struggled to accurately extract the specific text (evidence) that justified their safety decisions. Token-level F1-scores for evidence attribution were consistently low, indicating that while some models can guess a safety label, they lack the capability to reliably localize the specific unsafe content.

Furthermore, the study highlights that many models suffer from severe over-refusal (blocking safe content) or under-refusal (failing to block unsafe content), with few models achieving a balanced trade-off. The authors also demonstrate that Instruction Encryption (IE) attacks—where prompts are obfuscated—remain a major vulnerability for almost all tested models.