UNPREDICTABENCH: A Benchmark for Evaluating Distributional Randomness in LLMs | Amirhossein Abaskohi et al. | ResearchPod

Transcript: UNPREDICTABENCH: A Benchmark for Evaluating Distributional Randomness in LLMs

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a new paper that introduces a benchmark called UNPREDICTABENCH. The central claim is that while AI language models can *describe* randomness, they consistently fail to *generate* it in a way that matches real statistical patterns.

Alex: Can you unpack that distinction? Describing versus generating?

Sam: Sure. Imagine asking someone to explain how a fair coin works — they can tell you it lands heads half the time. But if you then ask them to *act like* a fair coin by calling out heads or tails a hundred times, they might unconsciously favour one side without realising it. That gap — between knowing how randomness works and actually producing it correctly — is exactly what this paper is testing.

Alex: So the question is whether these models can reliably stand in as simulators for systems that involve chance.

Sam: Precisely. And the concern is that models often collapse toward a single, plausible-looking answer instead of showing the true spread of possibilities. Think of it like a weather forecasting tool that always predicts partly cloudy — technically reasonable, but useless for capturing the full range of what weather can actually do.

Alex: That makes sense. It's like checking whether a dice roller is fair by rolling it a hundred times, rather than just looking at one roll.

Sam: That's a precise analogy. The researchers created 448 tasks, ranging from mathematics to code generation. For each task, they ask the model the same question 100 separate times, then compare that collection of answers against a reference set drawn from the true, known distribution.

Alex: And how do they decide if the model is actually doing well?

Sam: They use a statistical tool called the Kolmogorov–Smirnov test. Here's the intuition: imagine plotting two bar charts side by side — one built from the model's outputs, one from the real process. The test finds the single largest gap between those two shapes. If the gap is small enough, the model's outputs are statistically indistinguishable from the real thing.

Alex: So it's not about getting one correct answer — it's about whether the whole *collection* of answers has the right shape.

Sam: Exactly. And that turns out to be genuinely difficult for these models. The research points to the training process itself as a core reason why. When companies fine-tune models to be helpful and well-behaved — a process called alignment — they often, as a side effect, push the model toward giving the most common, expected response. That's useful for answering questions, but it quietly squeezes out the diversity that true randomness requires.

UNPREDICTABENCH: A Benchmark for Evaluating Distributional Randomness in LLMs

Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo, Ellie Dingqiao Wen, Lele Wang, Giuseppe Carenini, Peter West

Jun 4, 2026·3:37·arxiv·Original Paper

Distributional RandomnessUNPREDICTABENCHKS@N MetricBehavioral SimulationDistributional FidelityKS@N

Listen

4 min

Briefing

Abstract

We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems.

UNPREDICTABENCH: A Benchmark for Evaluating Distributional Randomness in LLMs: Research Paper Podcast

UNPREDICTABENCH: A Benchmark for Evaluating Distributional Randomness in LLMs

Listen