"Illusion Awareness" in CPS Design: Acknowledging That Assumptions Will Fail Is Safer Than Pretending They Won't

> Paper: *Towards Illusions Awareness in Cyber-Physical System's Design* > arXiv: 2609.17260 (Université Côte d'Azur / I3S / INRIA Kairos)

A Drone Story

A Crazyflie drone flew perfectly in the lab. It followed walls using optical-flow sensors to keep distance and an IMU to maintain attitude. All tests passed.

Deployed to a real warehouse—it crashed.

Why? The lab walls were white and matte; the warehouse walls were reflective metal. The optical-flow sensor failed on reflective surfaces, but the obstacle-avoidance logic still assumed "optical-flow readings are reliable." That assumption held in the lab but not in the warehouse.

The paper gives this phenomenon—"assumptions failing at runtime"—a precise name: design illusion.

What Is a "Design Illusion"

An illusion is neither a bug nor a failure. It is a state of persistently relying on an assumption that reality has falsified, while retaining false confidence in the system's correctness.

The definition has three key parts:

1. The assumption was reasonably made: the designer had reason to believe "optical flow is reliable" 2. It was falsified at runtime: the real environment proved it wrong 3. The design model still depends on it: the system has no mechanism to recognize "this assumption has failed" and keeps running on the original logic

Together, these form an illusion. What the designer "saw" was perfect lab performance; what was "real" was warehouse failure. The gap between the two is the reality gap.

The paper's core argument: the mainstream approach in CPS engineering is to "narrow the reality gap"—more realistic simulation, more precise sensors, more robust models. But that is not enough. We need another capability: when assumptions inevitably fail, the system should be able to recognize that it is in an illusion.

The Four Steps of Illusion Awareness

The paper proposes a methodology called Illusions Awareness, in four steps:

1. Identify

Use runtime monitoring to detect "candidate illusions" by comparing design-model expectations with actual observations. If the drone expects optical-flow readings in [0.5, 2.0] but reads 0.01, that is a candidate illusion.

But identification is only the first step—a single deviation may be noise. A repeated deviation pattern is needed to confirm an illusion.

2. Classify

Once confirmed, illusions are classified along two dimensions:

• Explicit vs. implicit assumptions: explicit assumptions are documented by the designer ("sensor works at 0–40°C"); implicit ones are dependencies the designer never consciously noticed ("walls are matte")
• Architectural layer: perception (sensors), computation (decision-making), actuation (control)

Classification makes illusions reasonable. Different types require different countermeasures.

3. Characterize

Each illusion is characterized along four dimensions:

• Origin: was the assumption derived deductively (from theory) or inductively (from data)?
• Validity: under what conditions does the assumption hold? Where are the boundaries?
• Confidence: how reliable is it? Based on how much data? What sensor quality?
• Impact: if the assumption fails, what happens? Drift? A crash? Mission failure?

These four dimensions form an illusion's "ID card," which determines how to handle it.

4. Leverage

The most crucial step: an illusion is not an enemy to eliminate, but a resource to exploit. Every identified and characterized illusion is design knowledge—it tells you "this assumption fails under these conditions."

For example, once the drone's optical-flow illusion is identified, the team can:

• Add an "optical-flow confidence" input to the avoidance logic and switch to ultrasonic mode when readings are anomalous
• Document that "the optical-flow assumption only holds on matte surfaces"
• Run dedicated pre-deployment tests on reflective surfaces

These are not "bug fixes" but conversions of illusions into design assets.

Why "Narrowing the Reality Gap" Is Not Enough

The mainstream CPS path is finer simulation, more robust models, more comprehensive testing—all narrowing the reality gap. But the paper points out a fundamental problem: the reality gap cannot be fully closed, for three reasons:

1. Open world: CPS operate in open environments; all scenarios cannot be enumerated 2. Sensor limitations: every sensor has blind spots and failure conditions 3. Time-varying conditions: environments change (day/night, seasons, new buildings)

If the gap is irreducible, "illusion awareness" becomes a necessary complementary capability. The goal is not to prevent illusions forever, but to let the system recognize and respond to them when they occur.

This matches how human experts behave. An experienced pilot does not assume "instruments are always accurate"—they cross-check instruments and enter a cautious mode on inconsistency. Experts are not error-free; they are aware when they err.

A Comparison with LLM Hallucinations

There is a striking structural parallel between CPS "design illusions" and LLM "hallucinations":

| Dimension | CPS design illusion | LLM hallucination | |---|---|---| | Root cause | Design assumption fails at runtime | Training distribution fails at inference | | Manifestation | System acts on a wrong assumption | Model generates from a wrong distribution | | Detection | Runtime monitoring of deviations | Output confidence estimation | | Response | Mode switching / degradation | Refusal / escalation |

Both face the same fundamental problem: how can a model trained in a closed world become aware of its limits in an open one?

Current LLM practice—training-time alignment (RLHF) and inference-time guardrails—parallels CPS's "narrowing the reality gap." What LLMs lack is "runtime illusion awareness": recognizing "I am hallucinating" during generation and switching modes. The framework-layer refusal mechanisms discussed in the CoSQ paper are essentially the LLM version of illusion awareness.

Limitations

The paper honestly acknowledges several limitations:

1. The framework is conceptual: no concrete implementation or quantitative evaluation 2. Incomplete taxonomy: there may be more illusion types than listed 3. Vague "leverage" step: directions are given, but no concrete method for turning illusions into assets 4. Only the Crazyflie case: applicability to other CPS domains (autonomous driving, industrial control) is untested

My Take: This Is "Assumptions as Debt," Engineered

Reading this paper, an analogy came to mind: design assumptions are like technical debt.

• Assumptions are reasonable when made (like borrowing to fund development)
• But environments change and assumptions fail (like accruing interest)
• Unmanaged, assumptions "default" (like debt blowing up)
• Illusion awareness is the "financial statement of assumptions"—how much you owe, when it matures, what default looks like

The problem in CPS engineering is not "having assumptions"—all engineering relies on assumptions. The problem is that assumptions are not explicitly managed. Designers write them into documents, deploy, and forget. At runtime, when assumptions fail, systems keep running on the original logic until something breaks.

The essence of "illusion awareness" is making assumptions first-class citizens—identified, classified, characterized, leveraged. Not eliminating assumptions, but making them manageable.

This applies beyond CPS. Any AI system operating in the open world—LLMs, robots, autonomous vehicles—needs some form of "illusion awareness." Acknowledging that assumptions will fail is safer than pretending they won't.

---

Paper link: https://arxiv.org/abs/2609.17260 Crazyflie project: https://github.com/geraldrocher/Crazyflie2.1