Defense against Poisoning Attacks under Shuffle-DP

> Paper: Defense against Poisoning Attacks under Shuffle-DP > Authors: Siyi Wang, Qiyao Luo, Yihua Hu, Lixu Wang, Quanqing Xu, Chuanhui Yang, Zhan Qin, Kui Ren, Wei Dong > arXiv: 2605.00625 | 2026-04-30

The Dangerous Assumption of Trusting Everyone

Consider a federated learning system where thousands of users contribute data, protected by shuffle-DP — the shuffle model of differential privacy, where data is shuffled and noised before reaching the central server. Shuffle-DP's core assumption: all users are honest.

In reality, some users may be malicious, sending fabricated data to distort the global model — this is a poisoning attack. The problem: shuffle-DP's design does not account for malicious users.

Why Shuffle-DP Is Vulnerable

Shuffle-DP is attractive because it offers a better privacy-utility balance than pure local DP (LDP) and stronger privacy than central DP. But its security assumptions are too idealized:

1. Privacy guarantees can break down — malicious users' fake data affects the post-shuffle distribution, so honest users' data may leak. 2. Utility is destroyed — poisoned data distorts statistics, making global analysis and models unreliable. 3. Detection is hard — the shuffle hides data provenance, so malicious users cannot be traced and accountability after the fact is impossible.

Defense Strategies: Detecting Poisoning While Preserving Privacy

Core challenge: how to detect and mitigate poisoning attacks while protecting privacy?

Key insight: poisoned data typically exhibits statistical anomalies, and even after shuffling and noise injection, anomalous patterns may persist as detectable residual signals.

Technical approaches discussed:

• Robust aggregation: use robust statistics (median, trimmed mean) instead of naive averaging to limit poisoned data's influence.
• Anomaly detection: detect anomalies in shuffled, DP-noised aggregates — requiring detectors designed for noisy data.
• Adaptive privacy budget allocation: increase privacy protection when anomalies are detected; reduce noise for utility when things look normal — dynamically balancing privacy and utility.
• User behavior modeling: even without individual attribution, model the population's behavior distribution and flag contributions deviating from it.

It is like detecting fraud in an anonymous vote: you cannot see who voted for whom, but you can check whether the statistics look anomalous.

The Privacy–Robustness Tension

• More privacy protection → more noise → harder to detect poisoning, easier for attackers to hide.
• More robustness → needs more real-data information → potentially less privacy for honest users.

> "Privacy protection makes poisoning detection harder, but detection needs more information — and more information means less privacy."

The paper explores finding balance in this tension: not all data is equally sensitive; some protections can be selectively relaxed, prioritizing sensitive information while allowing more scrutiny of less sensitive information.

Key Takeaway: Assume the Worst Case

Good security design assumes the worst case and still works under it. Shuffle-DP's assumption that all users are honest is a beautiful mathematical assumption — but the real world is not beautiful. A true privacy-preserving system must protect honest users even when malicious users are present.

Questions to ask if you build privacy-preserving systems:

1. Does my privacy scheme consider malicious participants? 2. Is there tension between privacy protection and robustness? 3. Am I providing security guarantees under idealized assumptions? 4. How would poisoning attacks affect my system?

Core lesson: privacy protection cannot be isolated from security considerations. Shuffle-DP is elegant in theory, but ignoring poisoning attacks makes it fragile in practice. Real privacy protection must work under adversarial conditions — protecting data from leakage *and* the system from destruction.

At the intersection of privacy and security, we need "both", not "either-or".