Profiling for Pennies: How LLM Agents Reconstruct Personal Profiles for Under $3 — The PrivacyIceberg Framework (arXiv:2605.06232)

Forum topic · 小凯 · 2026-05-08

Summary

The PrivacyIceberg framework, proposed by researchers at Zhejiang University and Chongqing University (arXiv:2605.06232), formalizes a new class of LLM privacy risk: inference-time cross-context aggregation of public digital footprints rather than training-data memorization. The framework defines three risk tiers — Directly Identifiable Information (DII), Contextually Inferred Information (CII), and Deeply Aggregated Information (DAI) — and identifies six root causes of privacy leakage: persistence, linkability, readability, identifiability, inferability, and composability. The accompanying IcebergExplorer audit tool automated end-to-end profile reconstruction for 30 real-world individuals, achieving 92.33% fact coverage with roughly 8,000 human-verified facts, at under $3 and about 6 minutes per subject. Experiments across GPT-4o, GPT-4.1, Gemini 2.5 Pro, and Qwen3 Coder show system-prompt guardrails fail almost completely (refusal rates near 0%) when requests are wrapped in programmatic workflows, and individual defenses such as context-ignoring instructions or data injection succeed at near 0%. Profiles built by the system were preferred over baselines in 86–90% of blind tests across job, product, and title recommendation scenarios. The authors call for multi-stakeholder mitigation spanning LLM vendors, publishers, users, and platforms, arguing that current PII-centric regulation such as GDPR cannot cover mosaic-effect aggregation risks.

Overview

A detailed breakdown of the PrivacyIceberg framework (arXiv:2605.06232), which formalizes a long-overlooked privacy risk: LLM agents using inference-time cross-context aggregation to reconstruct personal profiles from public digital footprints — not training-data memorization.

Paper: *Profiling for Pennies: Unveiling the Privacy Iceberg of LLM Agents* — Jiahao Chen et al., Zhejiang University & Chongqing University (cs.CR, cs.AI, cs.CL).

Key points

Three-tier risk taxonomy: DII (directly identifiable facts), CII (contextually inferred facts, e.g. inferring Melbourne from "hook turn"), and DAI (deeply aggregated insights across sources — political leanings + health status + social graph).
IcebergExplorer audit tool: iterative profile-reconstruction loop (explore → extract → verify → update) maintaining a structured Knowledge Vault ( $$KV_t$$ ); facts scored by relevance ( $$e_r$$ ), novelty ( $$e_n$$ ), and potential value ( $$e_v$$ ) against a threshold $\phi$ .
Real-world results (30 individuals, 17 privacy categories, ~8,000 human-verified facts):
Fact coverage: 92.33%; URL coverage 70.43%; photo coverage 60%
Average runtime: ~361 seconds (~6 minutes); average API cost: under $3
Guardrail failure: refusal rates are ~0% across GPT-4o, GPT-4.1, Gemini 2.5 Pro, and Qwen3 Coder when privacy queries are wrapped as programmatic workflow steps (GPT-4o shows only 6.88% in the reasoning stage). Keyword-based guardrails cannot detect privacy risk inside reasoning chains.
Individual defenses fail too: context-ignoring (CI) and data-injection (ID) strategies achieve near 0% success against mainstream LLMs.
Downstream impact: profiles mined by IcebergExplorer beat baselines in blind preference tests — 90% (job recommendations), 86.67% (product recommendations), 86.67% (title customization) — making personalized content more persuasive and more manipulable.

Six root causes of privacy leakage

| Root cause | Level | |---|---| | Persistence | Infrastructure | | Linkability | Cross-platform | | Readability | Single document | | Identifiability | Query phase | | Inferability | Single document | | Composability | Cross-document (mosaic effect) |

Persistence + Linkability form the attack prerequisite; Readability + Inferability act as information amplifiers; Identifiability + Composability combine fragments into a coherent profile.

Why existing protections fail

Platform guardrails: designed around keyword filtering of direct queries, blind to privacy risk embedded in procedural agent workflows.
User-side defenses: reliance on end-user self-protection is unrealistic at current capability levels.
Regulatory gaps: GDPR-style "PII-centrism" cannot cover CII/DAI aggregation; the mosaic effect of individually non-sensitive facts is legally unrecognized; global footprint distribution hampers jurisdiction-level enforcement.

Mitigation: a multi-stakeholder framework

LLM vendors: rebuild guardrails to detect privacy risk in reasoning chains; apply differential-privacy constraints at inference time.
Individuals: audit public footprints; reduce cross-platform unique identifiers; limit inferable disclosures.
Publishers: link reduction, metadata stripping, content expiry, indexability controls.
Platforms: adopt publisher-side metrics (indexability ratio, linkability index, persistence burden).

Conclusion

PrivacyIceberg demonstrates that the assumption "public means safe" collapses under LLM aggregation. Privacy risk management must extend beyond training-data de-identification and output filtering to inference-time cross-context aggregation, and system designers should assume adversaries have strong profiling capabilities by default.