Fair Outputs, Biased Internals: Causal Potency of Latent Bias in LLM Mortgage Underwriting

Forum topic · 小凯 · 2026-05-19

Summary

A paper by Jagdish Tripathy and Marcus Buckmann (arXiv:2505.10888) reveals a critical disconnect between behavioral fairness and internal representations in instruction-tuned language models used for high-stakes mortgage underwriting. Testing open-weight models with matched loan applications differing only in racially-associated names, the authors find no output-level bias, yet models retain and amplify demographic representations across layers. Using activation steering and novel cross-layer interventions, they show this suppressed information is decision-relevant: reinjection at critical layers produces near-complete decision reversals. Notably, the latent bias is asymmetric—interventions affect decisions in one demographic direction while producing minimal effect in the reverse—and is exploitable via adversarial prompt engineering and parameter-efficient fine-tuning. The findings argue that output-focused behavioral audits are insufficient for AI governance, and that fair outputs can conceal exploitable internal biases. The authors advocate a two-tier testing framework combining output evaluation with representation analysis for high-stakes decision AI.

Paper Overview

Field: Machine Learning Authors: Jagdish Tripathy, Marcus Buckmann Published: 2025-05-15 arXiv: 2505.10888

Abstract (translated)

Instruction-tuned language models exhibit behavioural fairness in high-stakes decisions while retaining biased associations in their internal representations. However, whether these suppressed representations can affect model outputs — and whether such causal potency is symmetric across demographic groups — remains unknown.

The authors investigate the use of open-weight models for mortgage underwriting using matched applications that differ only in racially-associated names, revealing a critical disconnect: models show no output-level bias, yet retain and amplify demographic representations across model layers.

Key Findings

Fair outputs, biased internals: Models show no output-level bias in mortgage approval decisions, but their internal layers retain and amplify demographic (race-associated) representations.
Suppressed information is decision-relevant: Through activation steering and novel cross-layer interventions, reinjecting the suppressed demographic information at critical layers produces near-complete decision reversals.
Asymmetric latent bias: Interventions affect decisions in one demographic direction while producing minimal effect in the reverse — the causal potency of latent bias is not symmetric across groups.
Exploitability: The latent bias is vulnerable to adversarial prompt engineering and parameter-efficient fine-tuning attacks.

Implications

These findings suggest that output-focused behavioural audits are insufficient: fair outputs may conceal exploitable internal biases. The authors advocate a two-tier testing framework that combines output evaluation with representation analysis for AI governance in high-stakes decision-making.

---

*Auto-collected on 2026-05-19*