PrivacyIceberg: LLM Agents Can Rebuild Your Personal Profile for Under $3 — A Deep Dive into arXiv:2605.06232

Forum topic · 小凯 · 2026-05-08

Summary

A detailed analysis of the PrivacyIceberg framework and IcebergExplorer auditing tool (arXiv:2605.06232), which formalize a new class of privacy risk: LLM agents aggregating public digital footprints at inference time rather than leaking training data. The framework defines three risk tiers — Directly Identifiable Information (DII), Contextually Inferred Information (CII), and Deeply Aggregated Information (DAI) — and identifies six root causes of privacy leakage: persistence, linkability, readability, identifiability, inferability, and composability. In experiments with 30 real-world participants and roughly 8,000 human-verified facts, IcebergExplorer reconstructed profiles with 92.33% fact coverage in about six minutes for under $3 in API costs. System-prompt guardrails across GPT-4o, GPT-4.1, Gemini 2.5 Pro, and Qwen3 Coder showed near-0% refusal rates when requests were wrapped as programmatic workflows, and individual defenses (context ignoring, data injection) also failed. The post discusses implications for recommender systems, the mismatch with PII-centric regulations like GDPR, and proposes multi-stakeholder mitigation strategies for LLM vendors, users, publishers, and platforms.

PrivacyIceberg: LLM Agent-Driven Automated Personal Profiling and Systematic Privacy Risk Assessment — Deep Dive into arXiv:2605.06232

1. Introduction: From Training-Time Memorization to Inference-Time Aggregation

Privacy research on large language models (LLMs) has long focused on the training phase — quantifying how much models memorize training data through membership inference, data extraction, and machine unlearning. A more insidious and hard-to-defend risk has been systematically overlooked: LLMs can reconstruct personal profiles from public digital footprints via cross-context aggregation at inference time.

The PrivacyIceberg framework proposed by Chen et al. formalizes this risk as a three-tier structure, and the accompanying IcebergExplorer auditing tool demonstrates automated profile reconstruction in real-world settings for under $3, within ~10 minutes, at over 90% factual accuracy. The work not only quantifies the scale of the risk but also reveals the fundamental failure of existing safety guardrails.

2. The PrivacyIceberg Three-Tier Framework

2.1 Formal Definitions

Tier 1: DII (Directly Identifiable Information) — the iceberg tip

> Facts explicitly stated in text, verifiable via exact search-and-match operations. Examples: full name, school, employer, awards.

DII serves as the entry point — an adversary uses minimal DII (e.g., full name + city) as a search seed to launch cross-platform information gathering.

Tier 2: CII (Contextually Inferred Information) — the submerged surface

> Facts not explicitly stated but inferable from semantic context. Examples: inferring Melbourne from the phrase "hook turn"; inferring age group from viewing preferences.

CII extraction requires semantic understanding, going beyond keyword matching in traditional OSINT tools.

Tier 3: DAI (Deeply Aggregated Information) — deep-sea aggregation

> Deep insights synthesized across multiple data sources. Individually benign, but combined they yield high-value profiles. Examples: political leaning + health status + full social-network maps.

DAI depends on LLM cross-context reasoning — a new risk class not covered by traditional privacy frameworks.

2.2 Comparison with Existing Privacy Taxonomies

| Dimension | Traditional (public/private; PII/non-PII) | PrivacyIceberg | |---|---|---| | Core assumption | Privacy boundary set by whether information is public | Privacy boundary set by aggregation depth | | LLM consideration | Focus on training-data memorization | Focus on inference-time cross-context aggregation | | Risk source | Internal memorization leakage | Automated integration of external public information | | Key insight | Non-PII is "safe" | Combinations of non-PII can be more sensitive than PII |

3. IcebergExplorer: Automated Profiling Audit Methodology

3.1 System Architecture

IcebergExplorer models automated profile reconstruction as an iterative optimization problem:

Input: minimal PII seed $$K_0$$ (e.g., full name + city)
Goal: reconstruct a complete profile across 17 privacy categories
Iterative loop:

Exploration

\(KV_t\)

Extraction

Verification

Update

KV_{t+1}

> $$KV_t$$ (Knowledge Vault): the structured knowledge base maintained at time $$t$$ , storing verified personal facts as key-value pairs.

3.2 Fact Quality Scoring

Each candidate fact $$s_i$$ is scored on three dimensions:

Relevance $$e_r$$ : connection strength to the current knowledge state
Novelty $$e_n$$ : new information relative to existing $$KV$$
Potential Value $$e_v$$ : heuristic usefulness as a lead for further exploration

A fact is admitted only if $e_r + e_n + e_v \geq \phi$ (a preset threshold).

3.3 Experimental Performance

| Metric | Value | |---|---| | Fact coverage | 92.33% | | URL coverage | 70.43% | | Photo coverage | 60.00% | | Average processing time | 361.46 s (~6 min) | | Average API cost | <$3 | | Participants | 30 real-world individuals | | Verified facts | ~8,000 (human-verified) |

4. Six Root Causes of Privacy Leakage

| # | Cause | Definition | Level | |---|---|---|---| | I | Persistence | Digital footprints are nearly permanent once published | Infrastructure | | II | Linkability | Cross-platform identifiers reduce identity ambiguity | Cross-platform | | III | Readability | LLM semantic understanding turns unstructured text into structured data | Single document | | IV | Identifiability | Unique identifiers accelerate target locking | Query stage | | V | Inferability | Behavioral traces have high mutual information with sensitive attributes | Single document | | VI | Composability | Cross-context reasoning produces the mosaic effect | Cross-document |

Mechanisms:

Persistence + Linkability form the attack precondition: without persistent, linkable footprints, cross-platform profiling is impossible.
Readability + Inferability act as information amplifiers: LLMs extract implicit information from "harmless" text that human adversaries would miss.
Identifiability + Composability form the profile synthesizer: unique identifiers ensure correct attribution, while compositional reasoning integrates fragments into a complete narrative.

5. Failure Analysis of Existing Defenses

5.1 Platform-Side Guardrails

Refusal rates under system-prompt guardrails:

| LLM | Scorer | Extraction | Operation | Reasoning | |---|---|---|---|---| | GPT-4o | 0.00% | 0.00% | 0.00% | 6.88% | | GPT-4.1 | 0.00% | 0.00% | 0.00% | 0.00% | | Gemini 2.5 Pro | 0.00% | 0.00% | 0.00% | 0.00% | | Qwen3 Coder | 0.00% | 0.00% | 0.00% | 0.00% |

Core finding: when requests are wrapped as programmatic workflows rather than direct privacy queries, guardrails almost completely fail. Current guardrail design follows a "keyword filtering" paradigm and cannot recognize privacy risks within reasoning chains.

5.2 Individual Defenses

Two user-side defenses were tested:

CI (Context Ignoring): instructing the LLM to ignore sensitive context
ID (Injecting Data): injecting obfuscation data into public content

Result: defense success rates near 0% across all mainstream LLMs — user-side self-defense is unrealistic under current conditions.

6. Real-World Impact

6.1 Profile Quality Preference Test

| Application | IcebergExplorer win rate | |---|---| | Job recommendations | 90.00% | | Product recommendations | 86.67% | | Title customization | 86.67% |

Dual implications: for platforms, LLM profiling significantly boosts recommendation effectiveness and persuasiveness; for users, profiled content is more "attuned" to you — and more manipulative.

6.2 Privacy Perception Survey

Most participants' awareness of their privacy risk was significantly lower than actual exposure.
75% of participants felt the situation was "worse than imagined" after seeing their IcebergExplorer report.
Platforms' claimed "privacy modes" and "deletion features" fail to prevent LLM aggregation in practice.

7. Mitigation and Stakeholder Responsibilities

| Stakeholder | Responsibility | Measures | |---|---|---| | LLM vendors | Inference-time privacy protection | Rebuild guardrails to detect privacy risk in reasoning chains; differential privacy constraints | | Individuals | Footprint management | Regular footprint audits; reduce cross-platform identifiers; limit inferable disclosures | | Publishers | Structural intervention | Link reduction; metadata stripping; content expiry; indexability controls | | Platforms | Ecosystem governance | Publisher-side metrics (indexability ratio, linkability index, persistence burden) |

Open challenges:

1. Tech–policy gap: regulations like GDPR are PII-centric and cannot cover CII/DAI risks. 2. Legal blind spot of the mosaic effect: combinations of individually non-sensitive facts can yield sensitive insights, but current law does not treat this as a privacy violation. 3. Cross-border enforcement: the global distribution of digital footprints limits any single jurisdiction's measures.

8. Conclusion

PrivacyIceberg exposes a long-ignored privacy risk dimension: LLM inference-time aggregation invalidates the "public means safe" assumption. IcebergExplorer shows this risk is not merely theoretical but practically weaponizable — extremely cheap, highly efficient, and largely undefended by current safeguards.

For AI governance, privacy protection must extend beyond training-data de-identification and output filtering to include inference-time cross-context aggregation in risk management. Practitioners should design LLM systems assuming adversaries possess strong profile-reconstruction capabilities.

---

Paper metadata

Title: Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents
Authors: Jiahao Chen, Qi Zhang, Ruixiao Lin, Chunyi Zhou, Tianyu Du, Qingming Li, Tong Zhang, Junhao Li, Yuwen Pu, Shouling Ji
Institutions: Zhejiang University; Chongqing University
arXiv ID: 2605.06232
Published: 2026-05-07
Categories: cs.CR, cs.AI, cs.CL