Not-quite-human tastes: the stylized omnivorousness of LLM survey surrogates

As market research firms increasingly turn to 'synthetic' survey panels generated by Large Language Models (LLMs), it is critical to determine if these models can accurately replicate human cultural tastes. This study evaluates the 'algorithmic fidelity' of silicon surrogates by comparing their responses to the Survey of Public Participation in the Arts (SPPA). The authors generated over 277,000 synthetic responses using models from OpenAI, Anthropic, and DeepSeek, assessing whether these models could capture the ecological distribution of tastes, the relational structure between genres, and the established associations between taste and social variables like age, class, gender, and race.

The researchers identified three primary failures in the fidelity of silicon tastes:

1. Systematic Positive Bias: Silicon surrogates exhibit a 'hyper-omnivorous' tendency, consistently over-reporting their liking for music genres compared to human respondents. This inflation is particularly pronounced for less popular genres.
2. Loss of Relationality: While human tastes are defined by complex, meaningful clusters (e.g., certain genres 'going together'), silicon samples fail to replicate these structures. They often dissolve genuine associations (like the link between blues and classic rock) while manufacturing spurious connections between unrelated genres.
3. Distorted Social Alignment: Silicon models fail to accurately map tastes to social space. They tend to over-associate tastes with youth, resurrect outdated class-based hierarchies of taste, and produce caricatured or entirely fabricated associations regarding gender and race.

The findings suggest that while LLMs can mimic the surface-level appearance of survey data, they lack the underlying sociological depth required to serve as reliable proxies for human cultural behavior. Using these models for market research or social science risks introducing significant, non-random biases that could lead to fundamentally flawed conclusions about public opinion and cultural consumption.