Towards Spec Learning: Inference-Time Alignment from Preference Pairs

Traditional methods for aligning large language models (LLMs) to user preferences typically involve either brittle, manual prompt engineering or computationally expensive gradient-based fine-tuning (like DPO). This paper investigates whether it is possible to automatically distill preference signals from a very small number of examples into a natural-language system prompt that steers the model effectively at inference time.

The authors propose 'spec learning,' a four-stage pipeline that processes a small set of preference pairs (e.g., 20 examples). First, a 'proposer' LLM generates candidate principles explaining why one response is preferred over another. These principles are then clustered and deduplicated (compression), validated against held-out data, and finally synthesized into a coherent system prompt (the 'spec'). This specification is then used to condition the base model during inference, requiring no parameter updates.

• Efficiency: The framework achieves competitive performance using only 20 preference pairs, often outperforming DPO models trained on 1,000 pairs.
• Interpretability: Unlike weight-based fine-tuning, the resulting specifications are human-readable, providing a transparent 'embodiment' of the preference signal.
• Scalability: While larger 'proposer' models yield slightly better specifications, the method remains highly effective even with smaller models, suggesting that the bottleneck is not model scale but the ability to extract and synthesize rules.
• Domain Sensitivity: The method excels in specialized domains (e.g., code security, math reasoning) where preferences are rule-based, but struggles with general-purpose datasets (like HH-RLHF) where preferences are too diverse to be captured by a small set of instructions.

Spec learning offers a practical alternative for developers who want to align models to specific behaviors without the infrastructure or data requirements of full fine-tuning. By shifting the alignment signal from opaque model weights to transparent, editable text, it provides a more controllable and interpretable path for model steering.