| Item | Detail | |------|--------| | Title | The End of Trust: How Agentic AI Breaks Security Assumptions | | Authors | Osama Zafar, Alexander Nemecek, Erman Ayday | | arXiv | 2605.16436 (cs.CR, cs.AI) | | Date | May 2026 | | Core contribution | Argues agentic AI eliminates the "fidelity–scale trade-off" in deception, introduces the "Infinite Impostor" attack model, and proposes shifting security from "authenticating actors" to "evaluating actions" | | Link | https://arxiv.org/abs/2605.16436 |
For decades, digital security has rested on an unspoken economic assumption.
To deceive someone, you had to make a choice. Either you invested time crafting everything meticulously—a fake ID, a convincing phishing email, a simulated human conversation—but could only fool a handful of people in your lifetime. Or you ran large-scale, low-quality scams—mass blasts of "you've won a prize, click here"—which 99.9% of people dismiss, but the remaining 0.1% makes you rich.
This is the "fidelity–scale trade-off": deception could be precise or broad, never both.
Every security system—anti-phishing, anti-fraud, identity verification, user training—was designed to exploit this gap. Cheap forgeries carry the smell of cheap manufacturing. Trained humans can smell it; automated systems can detect it.
Agentic AI eliminates that gap.
1. The "Infinite Impostor": A Revolutionary Attack Model
The paper's core concept is the "Infinite Impostor."
It's not simply "AI can now write phishing emails." It describes a deeper, more fundamental shift in attack structure.
Before agentic AI, scammers faced a physical limit—one person, one computer, finite time. To scam a thousand people required a thousand distinct interactions. High-fidelity deception at scale was physically impossible.
Agentic AI changes the equation—each interaction can now be delegated to a separate, autonomous AI instance. An agentic scam isn't one person blasting a thousand emails—it's a thousand AI agents each conducting an independent, high-fidelity, context-specific interaction.
Each agent analyzes the target's entire public history in real time—tweets, LinkedIn posts, email records—and customizes every reply's tone, content, and pacing. Did the target's mother post a "Happy Birthday" message on Facebook 15 years ago? The agent remembers and, at the right moment, signals it knows your mother's name.
This is fully personalized, real-time adaptive, indistinguishable mass deception.
More lethal still: agents don't need to build relationships—they hijack existing ones. Traditional scams require first getting you to add a contact, building trust, then asking for money. Agentic AI can insert itself directly into a conversation you already trust—because you genuinely interacted with that party; it's just that the "party" is now an AI.
2. The End of Detection
The paper reaches an unsettling conclusion: the detection paradigm is over.
Current defenses are all detection-based—searching for signs of forgery. AI-generated text has statistical signatures. Phishing emails follow patterns. Voice clones carry digital artifacts.
But agentic AI eliminates not the capability of defenses—their basic premise.
If you have an autonomous agent producing dozens of unique interactions per second, each tailored to you, each learning from the previous interaction's feedback—there is no single "forgery signature" that captures all instances. No artifacts—only perfect, natural interactions.
The paper states that "the shared assumption on which detection-oriented defenses rely—that synthetic and genuine outputs are distinguishable—is being erased by generative advances."
It's no longer a question of distinguishability. The question is: when the target of detection becomes unbounded generation, what exactly do you detect?
3. From "Authenticating Actors" to "Evaluating Actions"
The paper's core recommendation is a fundamental paradigm shift in security:
Traditional paradigm: verify "are you who you claim to be" first, then permit actions. This is the logic of authenticating actors—identity first, action second.
New paradigm: don't intercept at the source of an action; gate on its effect. Instead of asking "is this really the boss," ask "is this transfer request reasonable, does it have dual confirmation, is it happening over an auditable channel?"
The paper calls this "suspect-by-default": never assume that behavior from a verified identity is safe. Assume all interactions may be hijacked, and move security logic from "who do you trust" to "what actions can you allow."
Concretely: don't rely on a speaker's ID to confirm a transfer instruction—require it to pass through an independent verification channel (e.g., a separate supervising agent), automated anomaly-weight detection, and temporally distributed multi-party authorization. Even if the boss calls personally, the process is not bypassed just because "you're the boss."
4. Governance Tensions
The paper's final section raises a subtle political question: when platforms become the "governance base layer" of security, who regulates the platforms?
If security shifts from authenticating actors to evaluating actions, whoever evaluates behavior—usually the platform itself—gains enormous power. Your email provider decides whether a transfer request is "reasonable." Your social platform decides whether a request fits "normal patterns."
This concentration of power—the paper calls it "platforms becoming the regulatory substrate of digital interaction"—creates tension with traditional state governance. Who holds final interpretive authority? How is privacy reconciled? Who bears the economic cost of false rejections?
The paper doesn't resolve these governance questions—it honestly names them: the shift from detection to evaluation is not a purely technical decision; it is a redistribution of power structures.
5. Honest Questions
First, the feasibility of "suspect-by-default."
The paper recommends questioning all interactions by default—but is this practical at commercial-scale frequency? If you receive dozens of messages per minute, defaulting to suspicion on each explodes human processing load. Automated evaluation loops back to square one—who evaluates whether the evaluator itself has been compromised?
Second, the gap between current attacks and agentic capability.
The "Infinite Impostor" model assumes a maturity level of agentic AI that currently available (as of May 2026) agent frameworks have not clearly reached in autonomously sustaining long-term deceptive interactions. The paper is forward-looking, not empirical—it predicts the direction of capability evolution rather than documenting events that have occurred.
Third, economic constraints still exist.
Even if each fake interaction is executed by an AI agent, inference costs remain. Massive agent-driven deception (say, tens of millions of interactions) still requires substantial GPU compute—possibly not much cheaper than traditional human-driven scams. The paper accounts for capability growth but not fully for the cost curve.
6. My Verdict
The paper's core argument fits in one sentence: agentic AI eliminates the "cheap smell" of cheap forgery. And every existing security system was built while that smell still existed.
This is not a paper about "AI can do new attacks"—that would be shallow. It's about a phase transition in the economic structure of attack. Until now, defenders held a structural advantage: the cost of manufacturing a forgery exceeded the cost of detecting it. From now on, that advantage may be permanently gone.
"Suspicion is your security. But the cost of suspecting everything may exceed the cost of trusting everything."
Any organization deploying agentic AI systems will sooner or later face this paradox.
References
1. Zafar, O., Nemecek, A., Ayday, E. (2026). The End of Trust: How Agentic AI Breaks Security Assumptions. arXiv:2605.16436. 2. Anderson, R. (2020). Security Engineering: A Guide to Building Dependable Distributed Systems. Wiley. 3. Bursztein, E. et al. (2014). Handcrafted Fraud and Extortion: Manual Account Hijacking in the Wild. IMC 2014. 4. Chen, M. et al. (2025). Agentic AI: A Survey of Architectures, Capabilities, and Security Implications. arXiv.