Decentralized Aggregation of LLM Predictions via Wagering Mechanisms

How can multiple large language models (LLMs) with private data or domain expertise be aggregated in a decentralized setting, where aggregation weights must be determined without central oversight and must remain robust to strategic manipulation?

The authors propose WALLA (Advantage-Aligned Wagering Mechanisms for LLM Aggregation). In this framework, each model reports both a prediction and a wager. The wager acts as a stake reflecting the model's confidence in its prediction. The mechanism uses a leave-one-out baseline in the net payout function, which ensures that models are incentivized to report their true beliefs (incentive compatibility) and that their optimal wagers align with their expected comparative advantage over the rest of the pool. The authors instantiate two variants of this mechanism that trade off normality and no-arbitrage properties.

WALLA successfully enables decentralized learning of wager policies, allowing models to identify their own domain-specific strengths. Empirical results across question-answering (e.g., MMLU, PubMedQA) and forecasting benchmarks show that WALLA matches the performance of centralized aggregation methods (like stacked generalization) while being the only method that simultaneously achieves weighted aggregation, uncertainty awareness, and incentive compatibility. The learned wagers effectively filter out models that lack relevant context or domain expertise for a given query.

Existing decentralized aggregation methods often rely on heuristic confidence measures that are vulnerable to miscalibration or strategic exaggeration. WALLA provides a mathematically grounded mechanism that aligns the incentives of independent LLM services with the goal of high-quality collective prediction. By decoupling prediction from wager optimization, it allows smaller, specialized models to contribute effectively to an ensemble without needing to compete with generalist models across all domains.