Yanyi Pu, Damian A. Gonzalez-Salzberg, Zheng Yuan, Nikolaos Aletras
6 min
Predicting monetary remedies in legal settings is a complex task, especially when awards are determined on an equitable basis without explicit statutory formulas. This paper introduces ECtHR-NPD, a new benchmark designed to evaluate how well computational models can predict non-pecuniary damage (NPD) awards at the European Court of Human Rights (ECtHR). The dataset comprises 14,575 cases, providing a rigorous testbed for continuous monetary prediction. The authors evaluate a wide range of methods, including gradient-boosted trees, retrieval-based systems, fine-tuned encoder language models, prompted decoder language models, and knowledge-augmented agents.
The study finds that sophisticated language models and agentic approaches do not consistently outperform strong feature-based baselines like CatBoost. While models can sometimes capture the general ranking of awards, they struggle significantly with two critical aspects of the task: identifying cases that result in zero awards and accurately calibrating the magnitude of high-award cases. The results demonstrate that these models often fail to handle the heavy-tailed distribution of awards and the complexities introduced by multi-applicant or multi-violation cases. Furthermore, the performance of all models degrades sharply on the Challenging diagnostic subset, which includes complex cases like those heard by the Grand Chamber.
This research extends legal NLP evaluation beyond categorical classification and retrieval tasks into the domain of continuous monetary remedies. By exposing specific failure modes—such as zero-award recognition and magnitude calibration—the ECtHR-NPD benchmark provides a necessary tool for developing more robust and numerically aware legal AI. It highlights that current LLM architectures, which often treat numbers as token sequences, are not yet optimized for the specific demands of discretionary legal quantification.
Existing legal benchmarks cover diverse tasks, while continuous monetary remedies remain comparatively underexplored. We introduce ECtHR-NPD, to the best of our knowledge, the first benchmark for predicting non-pecuniary damage (NPD) awards at the European Court of Human Rights (ECtHR) from case information when no statutory formula or explicit calculation rule determines the amount. ECtHR-NPD contains 14,575 cases with case-level awards in nominal euros, chronological splits, and a protocol separating target construction from model input. We evaluate a battery of methods, including constant predictors, gradient-boosted trees, retrieval methods, fine-tuned encoder language models (LMs), prompted decoder LMs, and knowledge-augmented agents. Our results show that more sophisticated LM and agentic approaches do not consistently outperform the strongest feature-based baseline. All model families struggle to identify zero awards and to calibrate high-award predictions, with further degradation on the Challenging test view, making ECtHR-NPD a challenging testbed for current state-of-the-art open-weight and proprietary LMs.
Sam: And what about multi-applicant cases? If the model is predicting a single global amount, how does it handle cases where harm is distributed across different victims?
Alex: Poorly. The benchmark shows that models tend to scale predictions roughly linearly with applicant count — treating the award as a sum of individual harms. But the Court doesn't work that way. It takes a holistic view, issuing a global award that reflects the collective harm rather than aggregating individual entitlements. The model is doing arithmetic where the Court is doing equitable balancing, and the error rates in the Challenging diagnostic view reflect that gap directly.
Sam: So the failure isn't just numerical — it's a failure to capture the institutional logic.
Alex: Exactly. The model sees the case as a sum of parts. The Court sees it as a single, indivisible harm. That's not a calibration problem you fix by scaling the training data. It requires the model to internalize a different kind of reasoning.
Sam: Did structured inputs help at all? You could imagine that converting raw text into key-value features reduces the context burden and gives the model cleaner signal.
Alex: It reduces noise, but it doesn't solve the underlying problem. The bottleneck isn't parsing — it's the mapping from facts to currency. Structured features make the input cleaner, but the equitable inference step is still opaque to the model. [[RP_SECTION:evaluation-metrics-and-limitations|Evaluation metrics and limitations]]
Sam: Let me push on the evaluation design itself. These awards are discretionary, which means multiple values could be legally defensible for the same case. If a model is off by a few thousand euros but within the Court's historical range of variation, is that actually a failure?
Alex: That's the sharpest limitation in the paper. The point-error metrics measure fit to historical practice, not normative correctness. There's no single ground truth here — the Court's own awards reflect flexibility and context-sensitivity that a scalar error metric can't capture. A model that stays within the Court's historical variance is arguably capturing the latent logic, even if it misses the exact figure.
Sam: So we're measuring institutional consistency, not legal accuracy.
Alex: Right. And that distinction matters for how you interpret the results. High error doesn't necessarily mean the model reasoned badly — it might mean the Court itself would have accepted a range of values. The benchmark is a proxy, and a useful one, but researchers should be careful not to conflate alignment with historical precedent with alignment with legal correctness. [[RP_SECTION:future-research-directions|Future research directions]]
Sam: Where does the field go from here? The hurdle model addresses zero-inflation, but the OOD and Challenging views still show substantial error.
Alex: The likely next steps are number-aware loss functions — training objectives that penalize magnitude errors differently than directional errors — and better handling of distributional shift across case types. The benchmark's diagnostic structure is actually well-suited for that kind of targeted development, since it separates standard cases from procedurally complex ones. That lets you measure progress on the hard subpopulations without conflating them with the easier baseline performance.
Sam: It's a more honest evaluation framework than most legal NLP benchmarks, precisely because it forces the model to engage with the Court's discretionary logic rather than just classify outcomes.
Alex: And that's what makes it a meaningful contribution — not the headline error rates, but the diagnostic structure that reveals where and why current models break down. The point isn't to build a system that predicts awards to the euro. It's to understand what it would take for a model to reason the way an institution reasons. Thanks for listening to ResearchPod.