From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

This paper investigates whether the ALPHA benchmark's taxonomy-aware penalty—originally designed for evaluating Common Weakness Enumeration (CWE) vulnerability predictions—can effectively serve as a training signal for large language models (LLMs). While hierarchical penalties are theoretically useful for guiding models toward 'better mistakes' (errors that are semantically closer to the correct label), it remains unclear whether this signal can be successfully optimized during training without being distorted by the delivery mechanism.

The authors compare three distinct ways to deliver the ALPHA penalty to a Qwen2.5-Coder-7B backbone:

1. Supervised Fine-Tuning (SFT): Using the penalty only for evaluation while training on teacher-generated explanations.
2. Dual-Head Supervised Loss: Adding the penalty as a differentiable term to the cross-entropy loss in a classification head.
3. Reinforcement Learning (GRPO): Treating the normalized ALPHA penalty as a dense reward signal in a Group Relative Policy Optimization (GRPO) framework.

The researchers evaluate these methods on the Security Hardening and Adversarial Testing (SVEN) dataset, which represents a significant distribution shift from the training data (SecurityEval and CVEfixes).

The study finds that supervised approaches (SFT and dual-head loss) perform well in-distribution but regress below the zero-shot baseline when faced with distribution shift. In contrast, the GRPO approach demonstrates robust performance. The best GRPO policy reduced the cumulative ALPHA penalty on the SVEN dataset by 27.9% under greedy decoding, achieving statistical parity with a teacher model 4.5 times larger. The authors conclude that the effectiveness of a hierarchical penalty depends less on the metric itself and more on the directness of its delivery during optimization.