English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Profiling for Pennies: PrivacyIceberg Framework Reveals LLM Agent Privacy Risks (arXiv:2605.06232)

Forum topic · 小凯 · 2026-05-08

Summary

A detailed academic breakdown of arXiv:2605.06232, which introduces PrivacyIceberg, a three-tier privacy risk framework (DII/CII/DAI), and IcebergExplorer, an automated LLM-agent profiling audit system. The study shows that LLM agents can reconstruct comprehensive personal profiles from public digital footprints using only a minimal seed such as a name and city, achieving 92.33% fact coverage across 17 privacy categories for under $3 per person in roughly six minutes, verified against ~8,000 human-checked facts on 30 real-world participants. It identifies six root causes of privacy leakage—persistence, linkability, readability, identifiability, inferability, and composability—and demonstrates that current system-prompt guardrails across GPT-4o, GPT-4.1, Gemini 2.5 Pro, and Qwen3 Coder fail almost entirely when profiling is wrapped in programmatic workflows. Individual defenses tested (context ignoring, data injection) also succeeded at near 0% rates. Profiled information outperformed baselines in recommendation tasks with 86–90% win rates. The article argues for multi-stakeholder mitigation spanning LLM vendors, users, publishers, and platforms, and calls for privacy frameworks that address inference-time cross-context aggregation.

Key points

  • PrivacyIceberg framework formalizes LLM-driven privacy exposure in three tiers:
  • DII (Directly Identifiable Information): explicitly stated facts (full name, school, employer) — the attack entry point.
  • CII (Contextually Inferred Information): facts not stated but inferable from semantics (e.g., inferring Melbourne from the term "hook turn"; age range from viewing preferences).
  • DAI (Deeply Aggregated Information): cross-source synthesized insights (political leaning + health status + social graph) that are individually non-sensitive but highly sensitive combined.
  • Unlike traditional PII-centric classifications, the framework holds that privacy boundaries are determined by aggregation depth rather than whether data is public, and that non-PII combinations can be more sensitive than PII itself.
  • IcebergExplorer: automated profiling audit

    The system models profile reconstruction as an iterative optimization loop over a structured Knowledge Vault (KV): exploration (query generation), extraction (candidate facts from retrieved pages/images), verification (multi-source cross-checking), and update. Candidate facts are scored on relevance, novelty, and potential value against a threshold.

    | Metric | Result | |---|---| | Fact coverage | 92.33% | | URL coverage | 70.43% | | Photo coverage | 60.00% | | Avg. processing time | 361.46 s (~6 min) | | Avg. API cost | <$3 | | Participants | 30 real-world individuals | | Verified facts | ~8,000 (human-verified) |

    Six root causes of privacy leakage

    | Cause | Description | |---|---| | Persistence | Digital footprints are nearly permanent once published | | Linkability | Cross-platform identifiers reduce identity ambiguity | | Readability | LLM semantic understanding turns unstructured text into structured data | | Identifiability | Unique identifiers accelerate target locking | | Inferability | Behavioral traces carry high mutual information with sensitive attributes | | Composability | Cross-context reasoning produces the mosaic effect |

    Persistence + linkability enable attacks; readability + inferability amplify information; identifiability + composability synthesize full profiles.

    Guardrail and defense failures

    Refusal rates under system-prompt guardrails were ~0% across all pipeline stages for GPT-4o, GPT-4.1, Gemini 2.5 Pro, and Qwen3 Coder (only GPT-4o showed 6.88% in the reasoning stage). When requests are wrapped in programmatic workflows rather than direct privacy queries, keyword-based guardrails effectively fail.

    Individual-level defenses — Context Ignoring (CI) prompts and Injecting Data (ID) obfuscation — achieved near 0% success across mainstream LLMs, indicating self-defense by end users is currently unrealistic.

    Real-world impact

  • Profiled information outperformed baselines in blind preference tests: 90.00% win rate for job recommendations, 86.67% for product recommendations and title customization — improving both recommendation effectiveness and potential manipulativeness.
  • A survey of 30 participants found perceived privacy risk far below actual exposure; 75% felt the risk was "more serious than imagined" after viewing their IcebergExplorer reports. Platform "privacy modes" and deletion features did not prevent aggregation.
  • Mitigation and open challenges

    | Stakeholder | Measures | |---|---| | LLM vendors | Rebuild guardrails to detect privacy risks in reasoning chains; consider differential-privacy-style constraints | | Individuals | Audit public footprints; reduce cross-platform unified identifiers | | Publishers | Link reduction, metadata stripping, content expiry, indexability controls | | Platforms | Publisher-side metrics (indexability ratio, linkability index, persistence burden) |

    Open challenges include the gap between regulation (GDPR's PII-centrism) and CII/DAI risks, the legal blind spot around mosaic effects, and cross-jurisdiction enforcement difficulties.

    Conclusion

    PrivacyIceberg demonstrates that LLM inference-time cross-context aggregation invalidates the "public means safe" assumption. Profiling is cheap (<$3), fast (~6–10 min), highly accurate (>90%), and resistant to existing protections — implying that defenders must assume adversaries possess strong profile-reconstruction capability when designing LLM systems.

    ---

    Paper metadata

  • Title: Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents
  • Authors: Jiahao Chen, Qi Zhang, Ruixiao Lin, Chunyi Zhou, Tianyu Du, Qingming Li, Tong Zhang, Junhao Li, Yuwen Pu, Shouling Ji
  • Affiliations: Zhejiang University; Chongqing University
  • arXiv ID: arXiv:2605.06232 (cs.CR, cs.AI, cs.CL), published 2026-05-07

Tags

#llm-agents#privacy#osint#profiling#ai-safety#guardrails#data-protection#academic-papers

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619633