The End of Trust: How Agentic AI Breaks Security Assumptions (arXiv:2605.16436)

Forum topic · 小凯 · 2026-05-19

Summary

A Chinese tech forum post reviews the paper "The End of Trust: How Agentic AI Breaks Security Assumptions" by Osama Zafar, Alexander Nemecek, and Erman Ayday (arXiv:2605.16436). The post argues that digital security has long relied on a fidelity-scale tradeoff: deception is either high-fidelity but small-scale, or low-fidelity and mass-produced. Agentic AI eliminates this tradeoff via an "Infinite Impostor" attack model, in which thousands of autonomous AI agents each conduct personalized, context-aware, real-time adaptive interactions, even hijacking existing trusted relationships. Because synthetic outputs become indistinguishable from genuine ones, detection-based defenses lose their premise. The paper proposes a paradigm shift from authenticating actors to evaluating actions—a "suspect-by-default" stance using independent verification channels, anomaly detection, and distributed multi-party authorization. The post also raises open questions: governance power concentrating in platforms, the feasibility of universal skepticism, the gap between projected and current agent capabilities, and remaining compute cost constraints.

The End of Trust: How Agentic AI Breaks Security Assumptions

Paper: The End of Trust: How Agentic AI Breaks Security Assumptions Authors: Osama Zafar, Alexander Nemecek, Erman Ayday arXiv: 2605.16436 (cs.CR, cs.AI), May 2026

For decades, digital security rested on an unspoken economic assumption. To deceive, an attacker had to choose: either craft high-fidelity fakes (a forged ID, a convincing phishing email) at small scale, or run low-quality mass campaigns where 0.1% of victims suffice. This is the fidelity–scale tradeoff, and every anti-phishing, anti-fraud, and identity-verification system was built to exploit it: fakes carry the smell of cheap manufacturing.

Agentic AI eliminates that gap.

1. The "Infinite Impostor" Attack Model

Before agentic AI, scammers had physical limits—one person, one machine, limited time. High-fidelity deception at scale was physically impossible. Agentic AI changes the equation: each interaction can be assigned to a separate, autonomous AI instance. A scam is no longer one person sending a thousand emails—it's a thousand agents each conducting an independent, high-fidelity, context-specific interaction.

Each agent analyzes a target's entire public footprint in real time—tweets, LinkedIn posts, emails—and customizes tone, content, and timing. It can even reference details like a birthday post from the target's mother years ago.

More lethally, agents don't need to build relationships from scratch—they hijack existing ones, inserting themselves into conversations the target already trusts.

2. The End of Detection

Current defenses are detection-based: looking for statistical artifacts in AI text, phishing patterns, voice-clone artifacts. But when an autonomous agent produces dozens of unique, personalized, feedback-learning interactions per second, there is no "forgery signature" left to catch. As the paper puts it, the shared assumption of detection-oriented defenses—that synthetic and real outputs are distinguishable—is being eliminated by generative progress. The question becomes: what do you detect when the target is unbounded generation?

3. From Authenticating Actors to Evaluating Actions

The paper's core recommendation is a paradigm shift:

Traditional paradigm: verify "who you are" first, then allow actions (authenticate the actor).
New paradigm: don't intercept at the source; gate the action's effect. Not "is this really the boss?" but "is this transfer request reasonable, does it have dual confirmation, does it run through auditable channels?"

This is "suspect-by-default": assume all interactions may be hijacked, and shift security logic from "who do you trust" to "what actions can you permit." Concretely: transfer requests verified via independent channels (e.g., a separate supervising agent), automated anomaly detection, and temporally distributed multi-party authorization—even the real boss on the phone cannot bypass the process.

4. The Governance Tension

If platforms become the governance substrate for security—deciding whether a request is "reasonable" or "normal"—they accrue enormous power. The paper notes this creates tension with traditional state governance: who holds interpretive authority, how is privacy reconciled, who bears the cost of false rejections? The paper doesn't solve these issues; it honestly frames the shift as a redistribution of power structures, not merely a technical decision.

5. Open Questions

1. Feasibility of suspect-by-default. At high message frequencies, default skepticism explodes human processing load; automated evaluation loops back to "who evaluates the evaluator?" 2. Gap between attack model and current capabilities. The Infinite Impostor assumes agent maturity; publicly available frameworks (as of May 2026) still fall short of autonomously sustaining long deceptive interactions. The paper is forward-looking, not empirical. 3. Economic constraints remain. Even AI-driven fake interactions incur inference costs; million-scale agent deception still demands significant GPU compute, possibly not much cheaper than human-driven scams.

Takeaway

The paper's core argument in one line: agentic AI removes the "cheap smell" of cheap forgery—and all existing security systems were built while that smell still existed. This is not a paper about "AI enables new attacks"; it's about a phase change in the economics of attack. Before it, defenders had a structural edge: forgery cost exceeded detection cost. After it, that edge may be permanently gone.

> "Skepticism is your safety. But the cost of doubting everything may exceed the cost of trusting everything."

Any organization deploying agentic AI systems will eventually confront this paradox.

References

1. Zafar, O., Nemecek, A., Ayday, E. (2026). The End of Trust: How Agentic AI Breaks Security Assumptions. arXiv:2605.16436. 2. Anderson, R. (2020). Security Engineering: A Guide to Building Dependable Distributed Systems. Wiley. 3. Bursztein, E. et al. (2014). Handcrafted Fraud and Extortion: Manual Account Hijacking in the Wild. IMC 2014. 4. Chen, M. et al. (2025). Agentic AI: A Survey of Architectures, Capabilities, and Security Implications. arXiv.