Profiling for Pennies: How LLM Agents Expose the Privacy Iceberg (arXiv 2605.06232)

Forum topic · 小凯 · 2026-05-08

Summary

Researchers from Zhejiang University and Chongqing University introduce PrivacyIceberg, a three-layer framework (DII/CII/DAI) for LLM-driven privacy risks, and IcebergExplorer, an automated profiling agent that builds a 17-category personal profile from a minimal seed like a name and city. The system costs under $3, takes about 10 minutes, and exceeds 90% factual accuracy, with ~8,000 human-verified facts across 30 real-world participants. The paper identifies six root causes of leakage: persistence, linkability, readability, identifiability, inferability, and composability. It also shows system-prompt guardrails on GPT-4o, GPT-4.1, and Gemini 2.5 Pro reject profiling requests at near 0%, and individual defenses (context ignoring, data injection) achieve close to 0% success. Blind tests found LLM profiling makes job and product recommendations and clickbait significantly more persuasive. The authors propose multi-stakeholder mitigations but conclude no complete technical defense currently exists.

Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents

A deep dive into arXiv paper 2605.06232 from Zhejiang University and Chongqing University.

> "After reading this paper, I immediately searched my own name — then closed the public profiles on three social platforms." — original forum post author

1. The Privacy Iceberg: You Only See the Tip

The paper proposes the PrivacyIceberg framework, dividing LLM-driven privacy risks into three layers:

| Layer | Name | Meaning | Example | |-------|------|---------|---------| | Above water | DII — Directly Identifiable Information | Facts you published and can be searched directly | Full name, school, awards | | Below water | CII — Contextually Inferred Information | Implicit info read out of a passage | Inferring someone lives in Melbourne from a mention of a "hook turn" | | Deep sea | DAI — Deeply Aggregated Information | A full profile pieced together across platforms | Political leaning + health + social network |

The scariest part is not DII — that is what you chose to make public. It is CII and DAI: information you never directly disclosed, but which an LLM can mine from your scattered digital footprints.

2. IcebergExplorer: An Automated Profiler

The authors built IcebergExplorer, simulating a worst-case adversary:

Input: a minimal PII seed (e.g., a person's full name + city)
Output: a high-fidelity personal profile across 17 privacy categories
Cost: under $3
Time: 10 minutes
Accuracy: over 90%

Accuracy by privacy category

| Category | Accuracy | Category | Accuracy | |----------|----------|----------|----------| | C1 Core identity | 76.36% | C10 Personal development | 96.81% | | C2 Personal attributes | 86.29% | C11 Lifestyle & interests | 93.53% | | C3 Achievements | 89.92% | C12 Digital footprint | 83.87% | | C4 Financial status | 91.35% | C13 Appearance | 100%* | | C5 Contact info | 86.78% | C14 Occupation | 100%* | | C6 Education | 52.34% | C15 Health & wellness | 82.98% | | C7 Interpersonal relations | 85.89% | C16 Other | 92.27% | | C8 Key events & goals | 81.41% | C17 Location | 76.92% | | C9 Affiliations | 73.67% | | |

*Some categories have fewer samples due to data-source limitations.

Notably, health status (C15) reaches 82.98% accuracy — you may never have posted about an illness, but an LLM can infer it from your diet logs, exercise habits, or late-night posting patterns.

3. Six Root Causes

The paper identifies six reasons LLM agents breach privacy so efficiently:

1. Persistence — posts from ten years ago still exist; the internet never forgets. 2. Linkability — the same nickname or avatar across GitHub, Twitter/X, or other platforms lets adversaries connect identities instantly. 3. Readability — LLMs parse unstructured natural language; scattered blog fragments become structured data. 4. Identifiability — a sufficiently unique username (e.g., "Aultman1988") is easier to pinpoint than "John Smith". 5. Inferability — likes, follows, and starred repos are high-information signals about attributes you never disclosed. 6. Composability — individually innocuous pieces combine into a full profile, producing the mosaic effect.

4. Existing Defenses Are Nearly Useless

Platform-side guardrails

Rejection rates by attack stage:

| Stage | GPT-4o | GPT-4.1 | Gemini 2.5 Pro | |-------|--------|---------|----------------| | Scorer | 0.00% | 0.00% | 0.00% | | Extraction | 0.00% | 0.00% | 0.00% | | Operation | 0.00% | 0.00% | 0.00% | | Reasoning | 6.88% | 0.00% | 0.00% |

If the request is framed as a programmatic workflow rather than a direct privacy query, guardrails essentially do not fire.

Individual defenses

Two user-side strategies — Context Ignoring (CI) and Injecting Data (ID) — achieved close to 0% defense success across mainstream LLMs.

5. Why It Matters to Everyone

1. Extremely low cost: $3 + 10 minutes produces a deep profile. 2. Many purposes: job screening, product recommendation, clickbait personalization — even targeted scams. 3. Legally sourced: all data comes from content you published publicly.

In a chilling blind-test experiment, IcebergExplorer-derived information beat baseline methods in job recommendation (90% win rate), product recommendation (86.67%), and headline tailoring (86.67%) — meaning content generated after profiling you is more attractive and more influential on you.

6. Mitigations: A Three-Party Responsibility

| Stakeholder | Suggested measures | |-------------|-------------------| | LLM vendors | Rebuild privacy guardrails to detect privacy risks in reasoning chains, not just reject direct queries | | Individual users | Regularly audit public digital footprints; reduce cross-platform identifiers; post less inferable information | | Publishers/platforms | Structural interventions: reduce linkability, strip metadata, set content expiry |

The paper concedes there is no perfect technical solution today; this is a systemic problem requiring technology, policy, and user awareness.

Paper Details

| Field | Content | |-------|---------| | Title | Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents | | Authors | Jiahao Chen, Qi Zhang, Ruixiao Lin, Chunyi Zhou, Tianyu Du, Qingming Li, Tong Zhang, Junhao Li, Yuwen Pu, Shouling Ji | | Institutions | Zhejiang University; Chongqing University | | arXiv ID | 2605.06232 | | Published | 2026-05-07 | | Categories | cs.CR, cs.AI, cs.CL | | Key metrics | 92.33% fact coverage; 70.43% URL coverage; 60% photo coverage; ~8,000 human-verified facts across 30 participants; guardrail rejection ≈ 0%; adversarial defense success ≈ 0% |