CSPF: A Constrained Shared-Private Fusion Method for Non-Verifiable Preference Evaluation

Reliable evaluation of non-verifiable tasks—such as creative writing, open-ended dialogue, and subjective instruction following—is a persistent challenge in LLM alignment. Because these tasks lack ground-truth answers, researchers rely on reward models, rubric-based judges, or multi-evaluator systems. However, existing methods often collapse multidimensional human preferences into holistic scalar scores or rely on potentially misaligned natural-language rubrics. This paper investigates whether integrating hidden-state representations from multiple frozen reward models can provide a more expressive and accurate basis for preference assessment.

The authors introduce Constrained Shared-Private Fusion (CSPF). Instead of aggregating final scalar outputs, CSPF treats several frozen reward models as complementary experts. It extracts their hidden-state representations and projects them into a shared space. The architecture decomposes these signals into two components: a shared representation that captures common preference-relevant information across experts, and expert-private representations that preserve unique viewpoints. The fusion module is trained under pairwise human-preference supervision, using auxiliary constraints (Barlow Twins-style for shared features and supervised contrastive learning for private features) to ensure the representations remain structured and informative.

CSPF consistently outperforms single-expert reward models, rubric-based evaluators, and scalar-aggregation methods across both target-domain (LM-Arena) and out-of-distribution (PPE) benchmarks. Specifically, CSPF achieved the highest validation accuracy on LM-Arena (68.04%) and the best performance on the PPE off6 metric (61.67%). The results suggest that hidden-state fusion allows the model to leverage complementary signals from different experts more effectively than simply averaging their final scalar scores. Furthermore, the modular design allows for the integration of new reward models without retraining the underlying expert backbones.