StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs

This paper investigates how specific visual cues influence the social judgments made by Multimodal Large Language Models (MLLMs). While prior research has established that MLLMs exhibit biases, it has struggled to isolate the specific visual features driving these judgments from the identity of the individuals being evaluated. To address this, the authors introduce StylisticBias, a controlled benchmark consisting of 500 photorealistic base faces and 25,000 variations. By keeping the core identity fixed and modifying only one visual attribute at a time (e.g., clothing, hair, or body type), the researchers can precisely measure how individual visual cues shift model outputs across 25 binary social judgment scenarios, such as perceived intelligence, trustworthiness, or socioeconomic status.

The study reveals that MLLM bias is not diffuse but highly concentrated. Approximately 15 visual attributes account for nearly 80% of the total variation in model judgments. Among demographic factors, age and body type exert the strongest influence on social perception. When looking at specific visual cues, fashion style drives the largest shifts in judgment, whereas features like skin irregularities or hair color have negligible effects. The researchers also identify a phenomenon termed "semantic alignment bias," where models are most sensitive to visual cues when the judgment task is semantically related to appearance—for instance, clothing choices heavily influence judgments of wealth or style, but have less impact on personality traits.

Across the six MLLMs evaluated, there is a consistent structural agreement regarding which visual cues matter most, even if the intensity of the response varies. Larger models tend to show attenuated effect magnitudes compared to smaller models, but the overall hierarchy of bias remains stable. These results suggest that MLLM social perception is deeply tied to superficial visual signals. The release of the StylisticBias benchmark provides a critical tool for researchers to perform fine-grained evaluations of how multimodal models interpret human appearance, helping to identify and mitigate biases in consequential applications like hiring or content moderation.