Where Does the Noise Come From? A Variance-Components Decomposition of Non-Determinism in LLM Brand Answers

Researchers and commercial teams often face a reproducibility problem when using LLMs to measure brand sentiment: asking the same question multiple times yields different results. The common practice is to resample a prompt a few times and average the results, assuming that the noise is primarily due to the model's internal stochasticity. This paper challenges that assumption by decomposing the total variance of LLM responses into four distinct sources: within-prompt resampling, prompt paraphrasing, model identity, and query language.

Using a fully crossed dataset of 12,933 responses across 20 brands, 8 languages, 3 models, and 15 prompts, the author applies generalizability theory to partition the variance. By embedding these components into a decision-study framework, the paper calculates the optimal allocation of resources—how many languages, models, paraphrases, and repeats to purchase—to achieve a target level of measurement reliability.

The analysis reveals that query language is the dominant source of systematic variance (26.5%), far outweighing the brand identity itself (1.5%). This implies that a single LLM response is an extremely poor indicator of brand reputation. Once the variance is partitioned, within-prompt resampling accounts for 34.8% of the variance, and the interaction between the brand and its context accounts for 29.6%.

Crucially, the study demonstrates that the common practice of repeating a prompt is the least efficient way to improve reliability. The marginal reduction in error variance from adding a repeat beyond the fifth is nearly zero (0.0003). In contrast, spreading the query budget across different languages and models significantly reduces relative-error variance, providing a much higher return on investment for researchers seeking stable brand measurements.

This paper provides a rigorous, data-driven methodology for designing LLM-based measurement studies. By moving away from arbitrary "rule-of-thumb" resampling, researchers can optimize their query budgets to maximize the reliability of their findings. The work highlights that the "noise" in LLM outputs is not just random error but is deeply tied to the structural facets of the query design, such as language and model choice, which must be accounted for to produce trustworthy insights.