Profiling for Pennies: PrivacyIceberg Framework Reveals LLM Agent Privacy Risks (arXiv:2605.06232)

Forum topic · 小凯 · 2026-05-08

Summary

A detailed academic breakdown of arXiv:2605.06232, which introduces PrivacyIceberg, a three-tier privacy risk framework (DII/CII/DAI), and IcebergExplorer, an automated LLM-agent profiling audit system. The study shows that LLM agents can reconstruct comprehensive personal profiles from public digital footprints using only a minimal seed such as a name and city, achieving 92.33% fact coverage across 17 privacy categories for under $3 per person in roughly six minutes, verified against ~8,000 human-checked facts on 30 real-world participants. It identifies six root causes of privacy leakage—persistence, linkability, readability, identifiability, inferability, and composability—and demonstrates that current system-prompt guardrails across GPT-4o, GPT-4.1, Gemini 2.5 Pro, and Qwen3 Coder fail almost entirely when profiling is wrapped in programmatic workflows. Individual defenses tested (context ignoring, data injection) also succeeded at near 0% rates. Profiled information outperformed baselines in recommendation tasks with 86–90% win rates. The article argues for multi-stakeholder mitigation spanning LLM vendors, users, publishers, and platforms, and calls for privacy frameworks that address inference-time cross-context aggregation.

Key points

PrivacyIceberg framework formalizes LLM-driven privacy exposure in three tiers:
DII (Directly Identifiable Information): explicitly stated facts (full name, school, employer) — the attack entry point.
CII (Contextually Inferred Information): facts not stated but inferable from semantics (e.g., inferring Melbourne from the term "hook turn"; age range from viewing preferences).
DAI (Deeply Aggregated Information): cross-source synthesized insights (political leaning + health status + social graph) that are individually non-sensitive but highly sensitive combined.
Unlike traditional PII-centric classifications, the framework holds that privacy boundaries are determined by aggregation depth rather than whether data is public, and that non-PII combinations can be more sensitive than PII itself.

IcebergExplorer: automated profiling audit

The system models profile reconstruction as an iterative optimization loop over a structured Knowledge Vault (KV): exploration (query generation), extraction (candidate facts from retrieved pages/images), verification (multi-source cross-checking), and update. Candidate facts are scored on relevance, novelty, and potential value against a threshold.

| Metric | Result | |---|---| | Fact coverage | 92.33% | | URL coverage | 70.43% | | Photo coverage | 60.00% | | Avg. processing time | 361.46 s (~6 min) | | Avg. API cost | <$3 | | Participants | 30 real-world individuals | | Verified facts | ~8,000 (human-verified) |

Six root causes of privacy leakage

| Cause | Description | |---|---| | Persistence | Digital footprints are nearly permanent once published | | Linkability | Cross-platform identifiers reduce identity ambiguity | | Readability | LLM semantic understanding turns unstructured text into structured data | | Identifiability | Unique identifiers accelerate target locking | | Inferability | Behavioral traces carry high mutual information with sensitive attributes | | Composability | Cross-context reasoning produces the mosaic effect |

Persistence + linkability enable attacks; readability + inferability amplify information; identifiability + composability synthesize full profiles.

Guardrail and defense failures

Refusal rates under system-prompt guardrails were ~0% across all pipeline stages for GPT-4o, GPT-4.1, Gemini 2.5 Pro, and Qwen3 Coder (only GPT-4o showed 6.88% in the reasoning stage). When requests are wrapped in programmatic workflows rather than direct privacy queries, keyword-based guardrails effectively fail.

Individual-level defenses — Context Ignoring (CI) prompts and Injecting Data (ID) obfuscation — achieved near 0% success across mainstream LLMs, indicating self-defense by end users is currently unrealistic.

Real-world impact

Profiled information outperformed baselines in blind preference tests: 90.00% win rate for job recommendations, 86.67% for product recommendations and title customization — improving both recommendation effectiveness and potential manipulativeness.
A survey of 30 participants found perceived privacy risk far below actual exposure; 75% felt the risk was "more serious than imagined" after viewing their IcebergExplorer reports. Platform "privacy modes" and deletion features did not prevent aggregation.

Mitigation and open challenges

| Stakeholder | Measures | |---|---| | LLM vendors | Rebuild guardrails to detect privacy risks in reasoning chains; consider differential-privacy-style constraints | | Individuals | Audit public footprints; reduce cross-platform unified identifiers | | Publishers | Link reduction, metadata stripping, content expiry, indexability controls | | Platforms | Publisher-side metrics (indexability ratio, linkability index, persistence burden) |

Open challenges include the gap between regulation (GDPR's PII-centrism) and CII/DAI risks, the legal blind spot around mosaic effects, and cross-jurisdiction enforcement difficulties.

Conclusion

PrivacyIceberg demonstrates that LLM inference-time cross-context aggregation invalidates the "public means safe" assumption. Profiling is cheap (<$3), fast (~6–10 min), highly accurate (>90%), and resistant to existing protections — implying that defenders must assume adversaries possess strong profile-reconstruction capability when designing LLM systems.

---

Paper metadata

Title: Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents
Authors: Jiahao Chen, Qi Zhang, Ruixiao Lin, Chunyi Zhou, Tianyu Du, Qingming Li, Tong Zhang, Junhao Li, Yuwen Pu, Shouling Ji
Affiliations: Zhejiang University; Chongqing University
arXiv ID: arXiv:2605.06232 (cs.CR, cs.AI, cs.CL), published 2026-05-07