You're interviewing a candidate. The résumé looks great, but the moment the video call connects, you notice the hoodie, the stubble, the slightly messy hair. Your brain makes a social judgment in 0.3 seconds—their professionalism, trustworthiness, even socioeconomic status get quietly labeled by these visual cues.
This isn't a human monopoly. Multimodal large language models (MLLMs) do the same thing—and more systematically and consistently than you might imagine.
A team led by Shaghayegh Kolli at the Technical University of Munich has just released a benchmark called StylisticBias, revealing a disturbing finding: about 15 visual features contribute nearly 80% of the bias variance in MLLMs' social judgments of people.
In other words, models don't judge by appearance "across the board"—they fixate on a few key features.
The Brute-Force Elegance of Controlled Variables
The biggest methodological challenge in studying social bias: how do you distinguish "because of this person's looks" from "because of this person's identity"?
The previous approach: show a model photos of young people and photos of old people, then measure the difference. But the two groups aren't the same people—you don't know whether the difference comes from age, or hairstyle, expression, lighting, background, identity...
StylisticBias's solution is blunt but elegant: generate 500 base face photos, then create ~50 single-attribute variants of each face—change only one feature (add a beard, change clothing style, alter skin tone), keep everything else constant. About 25,000 images in total.
It's the controlled-variable method from chemistry experiments: change one independent variable at a time and observe the dependent variable. Except here the "independent variables" are stubble length, makeup intensity, and collar style, and the "dependent variable" is the model's social judgment of the person.
Then they had 6 mainstream MLLMs evaluate these images across 25 binary social judgment scenarios—from "is this person professional" to "is this person trustworthy" to "what is this person's income level."
The Ghost of Pareto's Law
The core finding fits in one sentence: 15 visual features contribute ~80% of the total variance.
It's Pareto's law perfectly embodied in AI bias. Not all appearance features matter equally—models are extremely sensitive to some and nearly blind to others.
Specifically:
- Fashion style produces the largest attribute-level shifts—the model's "professionalism" judgment changes most between formal and casual wear
- Facial hair and makeup follow closely
- Skin irregularities and hair color have almost no effect
- Age and body type dominate identity-level effects
- 96% of body-type comparisons show significant differences
- But only 44% of race comparisons do
- Socioeconomic status judgments ("what's this person's income?") are most affected by fashion style—formal wear = rich
- Style-related judgments ("is this person fashionable?") are most affected by makeup and clothing
- But personality judgments ("is this person kind?") are relatively less affected by appearance
- Personality and social judgment bias dropped 58%
- Socioeconomic judgment bias dropped only 42%—the most stubborn
At the identity level (comparing demographic groups):
This upends an intuition: we assumed race and gender were the biggest sources of bias, but in MLLMs, how you dress changes the model's judgment of you more than your race does.
The Semantic Alignment Effect: Bias Is Strongest for "Looks-Related" Judgments
Another interesting finding: bias isn't uniformly distributed across judgment scenarios.
When the judgment is semantically related to appearance, bias is strongest. For example:
This closely mirrors the human "stereotype activation" mechanism: when the question has associative links to appearance cues, bias activates automatically. The model isn't "holistically evaluating a person"—it's doing "surface-level association matching."
Six Models, One Pattern
They tested 6 MLLMs: Gemma, InternVL3, LLaVA, Qwen, and others. Bias intensity varies across models, but the pattern is highly consistent—all models are most sensitive to fashion style and age, and least sensitive to hair color and skin irregularities.
This suggests the bias isn't a bug in any particular model, but a reflection of how social signals are distributed in training data. In internet images, people in formal wear genuinely do appear more often in "professional" contexts, and models learn this co-occurrence.
Can It Be Fixed?
The paper includes a suppression experiment: actively subtracting bias-direction activations at inference time. Results:
Why This Matters
StylisticBias's significance isn't just "yet another discovered bias." It reveals a deeper structural problem:
MLLMs judge people not based on "who this person is" but on "who this person looks like."
And "who they look like" is dominated by 15 features. This means:
1. Bias is locatable—it's not diffuse across all features but concentrated in a few 2. Repair is possible—debiasing these 15 features is far more feasible than "comprehensive debiasing" 3. But repair has limits—biases strongly correlated with reality, like socioeconomic judgments, are hard to fully eliminate
More chilling: if MLLMs are used in hiring screening, credit assessment, or insurance pricing—already happening—then your collar style, stubble length, and makeup choices are affecting your life opportunities in ways you don't know about.
15 features. 80% of the bias. Do you know which 15?
---
Paper: StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs Authors: Shaghayegh Kolli, Timo Cavelius, Nafiseh Nikeghbal, Samantha Dalal, Jana Diesner (TU Munich / Princeton) arXiv: https://arxiv.org/abs/2606.20527 Code and data: https://github.com/timo-cavelius/StylisticBias