You are interviewing a candidate. The resume looks great, but the moment the video connects, you notice a hoodie, some stubble, slightly messy hair. Your brain completes a social judgment in 0.3 seconds — professionalism, trustworthiness, even socioeconomic status, all silently labeled by visual cues.
This is not a human monopoly. Multimodal large language models (MLLMs) do the same thing — more systematically and more concentrated than you might imagine.
A team at TU Munich led by Shaghayegh Kolli has just released a benchmark called StylisticBias, with a disturbing finding: about 15 visual features contribute nearly 80% of the bias variance in MLLMs' social judgments of people.
In other words, models don't judge by appearance "holistically" — they fixate on a few key features.
The Brute-Force Beauty of Controlled Variables
The biggest methodological challenge in studying social bias: how do you separate "because of this person's appearance" from "because of this person's identity"?
The old approach: take a set of photos of young people and a set of old people, have the model judge, and compare. But the two sets aren't the same people — you don't know whether differences come from age, or hairstyle, expression, lighting, background, identity...
StylisticBias's solution is crude but elegant: generate 500 base face photos, then create ~50 single-attribute variants of each face — changing only one feature (just adding a beard, just changing clothing style, just changing skin tone) while holding everything else constant. About 25,000 images total.
It's the controlled-variable method from chemistry class: change one independent variable at a time and observe the dependent variable. Except here the "independent variables" are stubble length, makeup intensity, collar style — and the "dependent variable" is the model's social judgment of that person.
Then they had 6 mainstream MLLMs evaluate these images across 25 binary social-judgment scenarios — from "is this person professional" to "is this person trustworthy" to "what is this person's income level."
The Ghost of Pareto's Law
The core finding fits in one chart: 15 visual features account for roughly 80% of total variance.
It's Pareto's law perfectly embodied in AI bias. Not all appearance features matter equally — the model is extremely sensitive to some and nearly blind to others.
Specifically:
- Fashion style produces the largest attribute-level shifts — formal vs. casual wear changes "professionalism" judgments the most
- Facial hair and makeup follow closely
- Skin irregularities and hair color have almost no effect
- Age and body type dominate identity-level effects
- 96% of body-type contrasts produce significant differences
- But only 44% of race contrasts produce significant differences
- Socioeconomic status judgments ("What is this person's income?") are most affected by fashion style — formal wear = rich
- Style-related judgments ("Is this person fashionable?") are most affected by makeup and clothing
- But personality judgments ("Is this person kind?") are relatively less affected by appearance
- Personality and social judgments: bias reduced by 58%
- Socioeconomic judgments: only 42% — the most stubborn
At the identity level (comparing demographic groups):
This overturns an intuition: we assumed race and gender were the biggest sources of bias, but in MLLMs, how you dress changes the model's judgment of you more than race does.
Semantic Alignment: Bias Is Strongest on "Seemingly Related" Judgments
Another interesting finding: bias is not uniformly distributed across judgment scenarios.
Bias is strongest when the judgment task is semantically related to appearance. For example:
This closely resembles human "stereotype activation": when the question you ask has associative links to appearance cues, bias activates automatically. The model isn't "holistically evaluating a person" — it's doing surface-level association matching.
Six Models, One Pattern
They tested 6 MLLMs: Gemma, InternVL3, LLaVA, Qwen, and others. Bias intensity differs across models, but the pattern is highly consistent — all models are most sensitive to fashion style and age, and least sensitive to hair color and skin irregularities.
This suggests bias isn't a bug in one particular model, but a reflection of the distribution of social signals in training data. In internet images, people in formal wear genuinely appear more often in "professional" contexts, and the model learned this co-occurrence.
Can It Be Fixed?
The paper includes a suppression experiment: actively subtracting bias-direction activations at inference time. Results:
Why This Matters
StylisticBias's significance goes beyond "yet another discovered bias." It reveals a deeper structural problem:
MLLMs judge people not based on "who this person is," but on "who this person looks like."
And "who they look like" is dominated by 15 features. This means: 1. Bias is locatable — not diffused across all features, but concentrated in a few 2. Repair is possible — debiasing these 15 features is far more feasible than "comprehensive debiasing" 3. But repair has limits — biases strongly correlated with reality, like socioeconomic judgments, are hard to eliminate entirely
More chilling: if MLLMs are used in hiring screening, credit assessment, insurance pricing — which is already happening — then your collar style, stubble length, and makeup choices are influencing your life opportunities in ways you don't know at all.
15 features. 80% of the bias. Do you know which 15?
---
Paper: StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs Authors: Shaghayegh Kolli, Timo Cavelius, Nafiseh Nikeghbal, Samantha Dalal, Jana Diesner (TU Munich / Princeton) arXiv: https://arxiv.org/abs/2606.20527 Code and data: https://github.com/timo-cavelius/StylisticBias