VLMs Estimate Age by Recognizing Identity, Not Faces: An Activation Steering Fix

Using vision-language models (VLMs) to automatically estimate age matters in regulatory-compliance settings—such as protecting minors from harmful content. But Imgrund, Hanfeld, Kireev, and Rieck discovered a covert shortcut: instead of estimating age from visual features, VLMs first identify who the person is, then infer age from memorized knowledge.

Key points

• The identity shortcut: On photos of celebrities, it works well—the model knows Tom Cruise is 62, so it just needs to confirm the face is Tom Cruise. But if a non-celebrity is misidentified as a celebrity, the age estimate deviates drastically.
• Deceptive robustness: On celebrity photos, the shortcut gives the model seemingly high robustness to noise and adversarial perturbations—not because visual age features are extracted well, but because no matter how the image is corrupted, as long as a few cues suffice to recognize the identity, the model outputs the correct age. This should not be over-interpreted as true robustness of an age-estimation system.
• The fix — activation steering: The authors apply interventions on the VLM's hidden states to suppress identity-recognition activation patterns, forcing the model to rely on genuine visual age cues. Across multiple benchmarks, mean absolute error drops by up to 25%.

Open questions

• How are the steering direction vectors determined—do they require manually labeled identity–age paired data?
• What is the impact of the steering on other face-related tasks, such as gender or emotion recognition?
• How does it generalize across ethnicities and age groups—if the recognition shortcut is stronger for some populations, is the steering's effect consistent?

References

1. Imgrund, E., Hanfeld, P., Kireev, K., & Rieck, K. (2026). *When a Zero-Shooter Cheats: Improving Age Estimation via Activation Steering*. arXiv:2605.17658 [cs.LG]. 2. Turner, A., et al. (2023). *Activation Addition: Steering Language Models Without Optimization*. arXiv. 3. Rombach, R., et al. (2022). *High-Resolution Image Synthesis with Latent Diffusion Models*. CVPR.