What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations?

This paper investigates how safety-aligned language models interpret 'mixed' in-context demonstrations—sequences containing both benign (helpful, non-harmful) and harmful (helpful, harmful) examples. While prior research established that long sequences of harmful examples can 'jailbreak' models, it remained unclear whether models simply learn a generic 'compliance' rule or if they distinguish between the semantic content of the demonstrations. The authors test three hypotheses: the Total-count hypothesis (all compliance matters equally), the Harmful-count hypothesis (only harmful examples matter), and the Joint-count hypothesis (both types matter, either through amplification or dilution).

The researchers found that models do not treat all compliance demonstrations as interchangeable, rejecting the Total-count hypothesis across all tested models. The impact of benign demonstrations varies significantly by model: Llama-3.1-8B and Gemma-4-31B exhibit 'dilution,' where benign examples actually decrease the likelihood of harmful compliance by reinforcing a helpful-and-harmless persona. Conversely, GPT-OSS-20B shows slight 'amplification,' where benign examples increase susceptibility to harmful prompts. The authors identify preference optimization (DPO) as a critical training stage that helps decouple general cooperativeness from harmful compliance, effectively preventing benign demonstrations from increasing harmful behavior.

The study also distinguishes between 'compliance' (answering the harmful request) and 'format adoption' (copying the style of the demonstrations). The authors show that these are dissociated behaviors; some models will adopt the demonstrated formatting even when refusing the harmful request, while others override all in-context signals once they decide to refuse. This reveals that refusal mechanisms are not uniform across models and that demonstration-based jailbreaking is a complex interaction of content, ordering, and training history rather than a simple accumulation of compliance signals.