From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation

Large language models (LLMs) are increasingly used for quantitative prediction tasks in high-stakes fields like healthcare and finance. However, these models are prone to hallucinations and overconfident errors. This paper addresses the critical need for models that not only provide accurate numerical estimates but also output reliable confidence signals, allowing users to know when a prediction is trustworthy.

The authors introduce CARE-PPO (Confidence-Aligned Reward for Estimation with PPO), a reinforcement learning framework that connects loss prediction theory with actor-critic fine-tuning. The core innovation is the use of a Confidence-Aligned Reward for Estimation. By defining the reward as a monotonic function of the model's prediction error, the training process provides dense, error-aware feedback to the actor. Simultaneously, the critic—which is trained to estimate expected rewards—naturally learns to map states to expected prediction quality. During inference, the critic's value function is repurposed as a confidence score, eliminating the need for post-hoc confidence calibration or explicit confidence supervision.

CARE-PPO was evaluated on nutrition (carbohydrate) estimation and product price prediction using Qwen-3 models. The results demonstrate that CARE-PPO significantly improves confidence alignment compared to standard logit-based and verbalized confidence baselines. Furthermore, the framework achieves strong quantitative prediction performance (measured by Mean Absolute Error) while reducing task-specific overfitting on general instruction-following prompts. The authors show that these gains are robust across different domains and linguistic shifts, outperforming binary reward-based reinforcement learning methods.

This work provides a practical, integrated solution for deploying LLMs in safety-critical quantitative tasks. By embedding confidence estimation directly into the reinforcement learning loop, the authors offer a way to make LLM outputs more interpretable and reliable without sacrificing the model's generative capabilities or requiring external, post-hoc calibration tools.