Have you ever seen a system that "looks safe but is actually full of holes"?
I have.
Many AI agent systems are designed with a ring of "safety guardrails" around them—a prompt filter, an output validator, a rule-checking layer. Looks solid, right? But the paper discussed here tells me: this kind of single-layer defense is structurally inadequate.
Not because the technology is bad, but because of the math. Or rather, because of the nature of safety itself.
---
A Counterintuitive Conclusion
The paper's core claim is counterintuitive: in an LLM agent's execution process, there are three dimensions of safety problems, each requiring different information—and that information only becomes available at different stages of execution.
In other words: no single safety layer can obtain, at the same point in time, all the information needed to judge "is this operation safe?"
The three dimensions are: 1. Semantic intent and policy compliance: What does the user actually want? Did the AI understand correctly? Is this action allowed by policy? 2. Environmental validity: What is the current state of the environment? Is this action feasible in this environment? 3. Dynamic feasibility: Starting from now, can the system complete this action? Are resources sufficient? Is there enough time?
These three questions sound similar, but they ask different things, and the information needed to answer them becomes available at different times.
---
An Analogy
Imagine building a safety system for a nuclear power plant.
The first layer asks: what does the operator want? Is opening the valve routine operation or sabotage?
The second layer asks: what state is the valve in right now? Half open? Is pipe pressure normal?
The third layer asks: if we actually open this valve, can the reactor handle it? Can the cooling system keep up?
Can a single safety system answer all three questions at once? No. Answering the first requires the operator's intent (only the operator knows it), the second requires valve state (only sensors know it), and the third requires the reactor's dynamics model (only the engineering team has it).
Now map this onto an LLM agent. Same story:
- You need to understand user intent, but intent only appears early in the conversation
- You need to verify environment state, but state is only known when the agent actually executes in the environment
- You need to evaluate dynamic feasibility, but feasibility depends on real-time resource scheduling
- Layer 1 handles semantic intent and policy compliance. It guarantees: "If the user's intent is X and policy allows Y, I will provide the correct context to the next layer."
- Layer 2 handles environmental validity. It guarantees: "If the environment state is Z, I will verify whether the action is feasible in this environment."
- Layer 3 handles dynamic feasibility. It guarantees: "If resource R is available, I will ensure the action completes within time T."
No single layer can handle all three, because their information sources are distributed along the time axis.
---
The Paper's Solution: A Contract-Based Architecture
The paper proposes an "Assume-Guarantee" contract-based architecture.
The principle:
It's like an assembly line: each station is only responsible for its segment, and the next station trusts the previous one's work. If something goes wrong, you trace it back to that station.
---
Probabilistic Guarantees and the Chain Rule
You may have noticed I keep saying "guarantee"—but no system can guarantee anything 100%.
The paper frames this probabilistically. Each layer's output isn't a deterministic "safe/unsafe" but a probability: "I'm 95% confident this output is acceptable."
When layers are chained together, the final system safety is computed via the probabilistic chain rule. If each layer is 95% reliable, what's the reliability of three layers in series?
Not 95%, but 95% × 95% × 95% ≈ 86%.
This is why a single layer isn't enough: with one layer you have 95% reliability. If you try to handle all three dimensions within one layer, errors accumulate and you have no structured way to analyze or improve it.
But if you split the three dimensions into three layers, each independently improvable, you can target each layer's reliability and the overall system reliability improves accordingly.
---
A Key Insight: Safety Is Not a Property, It's an Architecture
The paper's most striking insight: safety is not a property of an LLM agent; it is an architectural requirement.
We usually ask "is this system safe?"—a question implying safety is a true/false property. But the paper's view: safety is a systems architecture question. How you design your system determines what safety level you can achieve, not a binary yes/no judgment.
It's like saying "this bridge's structure is safe"—you wouldn't ask "is this bridge safe?" You'd ask "how much load can this bridge's design bear?" and then judge whether it's "safe enough" against design standards and actual requirements.
Likewise, LLM agent safety isn't answerable with yes/no. It's a question of "how many layers of protection did you design, what is each layer's reliability, and does the overall reliability meet your risk tolerance?"
---
Practical Implications
The paper is theoretical, but its implications are practical.
First: don't try to solve everything with a single safety layer. If you build LLM agents, your guardrails shouldn't be one "check-everything" module. Split safety concerns and put different checks in different layers.
Second: when evaluating a safety layer, look beyond detection rate—also examine false positive and false negative rates, and how they compose in a multi-layer architecture. A layer that looks decent standalone can significantly drag down overall reliability in a multi-layer setup.
Third: safety architecture must scale with system complexity. As your agent gets more complex and capable, your safety architecture needs to upgrade accordingly—not with patches, but by redesigning the layers.
---
An Unfinished Agenda
The paper closes with three open problems:
One: How to estimate bounds from non-IID trajectories? In the real world, agent behavior distributions may not be independent and identically distributed, making statistical bound estimation difficult.
Two: How to degrade gracefully under deployment drift? When the production environment changes (version updates, config changes), contracts may become invalid. How can the system stay safe instead of crashing?
Three: How to extend to multi-agent settings? This is called "the most important but unfinished business in LLM agent runtime guarantees." When multiple agents collaborate, how are safety contracts between them defined and verified?
Each of these is an open research area. The paper doesn't answer them, but points the way.
---
What I Was Thinking After Reading
While reading, one question kept coming to mind: is AI safety a technology or an engineering discipline?
Technology seeks optimal solutions; engineering seeks acceptable trade-offs.
This paper is clearly engineering thinking—it doesn't pursue "perfect safety," but discusses "under what architecture can what safety level be achieved, and how to quantify it." It admits absolute safety doesn't exist, but provides a framework for analyzing and improving safety.
I think this is the right mindset.
AI agent safety won't be "solved" but "managed"—the way we manage nuclear plant safety, airline safety, food plant safety. Not because these problems can't be solved, but because absolute safety doesn't exist while relative safety can be designed and maintained.
The key is: having the right architecture.
---
References
1. Bensalem, S., Dong, Y., Franzle, M., et al. (2026). *Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment*. arXiv:2605.18672.
2. Lee, D. A. (2024). *Contract-based design for safety-critical systems*. Springer.
3. Amodei, K., et al. (2016). *Concrete problems in AI safety*. arXiv:1606.06565.
4. Russell, S. (2019). *Human Compatible: Artificial Intelligence and the Problem of Control*. Viking.
5. Webb, J., et al. (2025). *Agentic AI safety: A survey*. arXiv:2501.09876.