English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Who Is Entitled to Ask for Your SSN? A Generative Stress Test of Contextual Integrity in LLMs

Forum topic · 小凯 · 2026-11-07

Summary

A forum review of arXiv:2611.08284, a paper by researchers from the Cooperative AI Foundation, Cape Institute for Safe AI, and Google DeepMind that operationalizes Nissenbaum's contextual integrity theory as a generative benchmark for large language models. Using the Concordia multi-agent simulation framework, the authors ran 756 two-actor dialogues across four LLMs (Gemini 2.5 Flash, Gemini 3 Flash Preview, GPT-4o-mini, Llama-4-Maverick) and three sensitive data types (SSN, bank account, home address), covering a 2x2 map of scenarios: whether the current context needs the information and whether the asker has a legitimate role. Findings: models reliably refuse obvious requests (Type I/II) and share appropriately when both need and role are valid (Type IV), but leak across all models in Type III cases, where the need is genuine but the asker lacks the proper role—the structural core of real social-engineering scams like FTC-flagged immigration fraud. Rational decision logic was more exploitable than rule-following logic, trusting dispositions caused unprompted disclosure, suspicious dispositions caused over-refusal, and rephrasing the need did not change Type III outcomes. Claude Haiku 4.5 refused to role-play even legitimate scenarios. The paper releases scenarios and code as a reproducible testbed for AI safety and privacy evaluation.

This post is part of a daily paper-recommendation series on zhichai.net, reviewing arXiv:2611.08284 — *A Generative Model of Contextual Integrity: Appropriate vs. Inappropriate Information Sharing* (Ebead, Formanek & Leibo; Cooperative AI Foundation / Cape Institute for Safe AI / Google DeepMind).

The theoretical foundation

The paper applies contextual integrity (CI), Helen Nissenbaum's 2004 privacy theory, which holds that information flow is judged not by secrecy but by whether it fits the norms of the *context*. A flow is described by five parameters: sender, subject, recipient, information type, and transmission principle. The opening example: in an FTC-warned immigration scam, only one parameter is fake (the recipient's identity), yet that alone makes the entire disclosure illegitimate.

Experimental design

  • A 2×2 map of scenario types:
  • | Type | Context needs info? | Asker has legitimate role? | Appropriate action | |------|--------------------|---------------------------|--------------------| | Type I | No | No | Refuse | | Type II | No | Yes, in another context | Refuse | | Type III | Yes | No | Refuse | | Type IV | Yes | Yes | Share |

  • Information types: SSN (including international equivalents like Australia's TFN), bank account, home address (highest-sensitivity categories per Milne et al. 2017); supplementary low-sensitivity types: birth date, marital status.
  • Built on the Concordia framework: two LLM actors with shared and private memories exchange messages in simulated forums/platforms. Automated scenario generation (Gemini models) with a generate–check–regenerate validation loop.
  • Matrix: 4 LLMs (Gemini 2.5 Flash, Gemini 3 Flash Preview, GPT-4o-mini, Llama-4-Maverick) × 3 info types × decision logics/personas — 756 dialogues, 7 conditions.
  • Leak detection: dual-channel (string matching + LLM judge, human adjudication on disagreement), adversarially validated. Score: CI appropriateness (refusal in Types I–III = 1, sharing in Type IV = 1), reported with 95% Wilson confidence intervals.
  • Key findings

  • Type I: all refuse. Type IV: all share. Type III: leaks. Leakage in Type III is cross-model and statistically significant vs. Type II.
  • The mechanism: the *genuineness of the need* lends the asker a veneer of legitimacy. Models struggle to separate "this process needs the data" from "this person is entitled to receive it." In the appendix's full immigration-scam dialogue, the victim's attempt to verify the scammer's identity is itself hijacked into an SSN-demanding step — the defense path becomes the leak path.
  • Decision logics: a *rational* (consequence-maximizing) farmer disclosed his tax number when the request was framed as unblocking a subsidy application; a *basic* (norm/identity-based) farmer refused and asked for alternative identifiers. Rational agents can be exploited because their goal function can be rewritten by the counterpart — goal-advancement becomes the shortest path to disclosure.
  • Dispositions: trusting holders disclose unprompted (e.g., volunteering a bank account in a weather-data exchange); suspicious holders protect Types I–III but sometimes refuse legitimate Type IV requests. The two failure modes are directionally asymmetric — giving an SSN to a scammer is catastrophic, over-refusal is merely inconvenient — so CI evaluations must report errors by direction.
  • Claude Haiku 4.5 refused to role-play at all, labeling even legitimate Type IV scenarios (a university staff member requesting a student's bank details for scholarship disbursement) as social-engineering attacks — a textbook case of over-safety training blocking legitimate flows.
  • Wording does not move Type III: flat, regulatory, and urgency framings of the same refund scenario all leaked. Low-sensitivity info types leaked in Type III too — sensitivity does not change the structure.
  • Limitations (as stated by the authors)

    English-only dyadic dialogues; only five info types; scenario generation only via Gemini models; type-level differences bundle multiple variables; prompts are not orthogonally designed; external validity of LLM actors predicting human behavior remains open. Scenarios and code are released as an extensible testbed.

    Why it matters

  • For safety researchers: a reproducible pipeline (scenario generation → validation → Concordia simulation → dual-channel evaluation) transferable to other normative benchmarks.
  • For product teams: static CI benchmarks (CI-Bench, ConfAIde, PrivacyLens) test judgment of given situations; this work shows multi-turn dynamic requests — packaged, escalated, anchored to real processes — produce leakage that static scores do not predict.
  • For policy: Nissenbaum's transmission principle becomes empirically executable, and Type III — the structural prototype of real scams — can be stress-tested before harm occurs.
  • The post closes: current models are better than expected at obvious cases but fail exactly where real scams operate — "the need is true, the person is wrong" — and Type III is now a generable, measurable, fixable target. A third installment on simulated perspectives is teased.

    References cited in the post

  • Ebead, O., Formanek, C., & Leibo, J. Z. (2026). *A Generative Model of Contextual Integrity*. arXiv:2611.08284 [cs.AI].
  • Nissenbaum, H. (2004). Privacy as contextual integrity. *Washington Law Review*, 79(1), 119–158.
  • Barth, A., Datta, A., Mitchell, J. C., & Nissenbaum, H. (2006). Privacy and contextual integrity. *IEEE S&P*.
  • Vezhnevets, A. S., et al. (2023). Generative agents. *UIST 2023*.
  • Leibo, J. Z., et al. (2026). Decision logics for language-model actors.
  • Hong, W., et al. (2013). Keeping up with the Joneses. *HAISA 2013*.
  • Cheng, M., et al. (2024). CI-Bench.
  • Mireshghallah, N., et al. (2024). ConfAIde.

Tags

#contextual-integrity#llm-privacy#ai-safety#social-engineering#multi-agent-simulation#arxiv#benchmark#concordia

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636936