When Higher Observation Fidelity Hurts Problem Solving: Why AI Robots Get Worse with Better Sensors

> A translated and annotated discussion from zhichai.net of *When Higher Observation Fidelity Hurts Problem Solving* by Oussama Zenkri and Oliver Brock (arXiv:2605.20072, 2026).

The Opening Metaphor: Microscope and Lost Traveler

Imagine driving somewhere new. One navigation app says: "Turn right in 200 meters." Simple and clear. Now imagine a second app simultaneously broadcasting: "In 47 meters there's a pothole 3 cm in diameter, the road surface changes from asphalt to concrete, two pedestrians on the right sidewalk approaching at 1.2 m/s, suggest reducing speed by 8% and adjusting the steering wheel by 2.3 degrees..."

You would most likely miss the turn. Excess detail isn't help—it's cognitive noise. This post opens with that analogy to explain the paper's striking finding: giving AI robots higher-fidelity sensor data can make them perform worse.

Background: Embodied LLMs

Researchers increasingly place large language models (LLMs) into robot bodies. These "embodied LLMs" take in sensor data—camera images, LiDAR, tactile feedback—and plan actions through natural-language reasoning. The traditional intuition: better sensors → deeper environmental understanding → better decisions. Like giving a nearsighted person better glasses.

This paper argues the opposite: there is an information-overload threshold beyond which more detail equals worse performance.

What Is Observation Fidelity?

Observation fidelity is the precision and richness of sensor data:

• Low fidelity: "There is an obstacle ahead"
• Medium fidelity: "A red cube, roughly 30 cm per side, 2 meters ahead"
• High fidelity: "A cube with RGB (220, 45, 32), 2.03 meters ahead, side length 29.7 cm, surface roughness 0.8, reflectivity..."

Sensor data (point clouds, depth maps) must be encoded before entering an LLM—via ViT patch encoding, depth-map projections, or dedicated vision encoders. But an LLM's context window is finite: as sensor data grows more granular, it crowds out context space that should be spent on reasoning.

Core Findings

The authors systematically compared LLM performance across embodied problem-solving tasks at different observation fidelities:

| Task | Low Fidelity | Medium Fidelity | High Fidelity | |------|---------|---------|---------| | Navigation | ✅ Success | ✅ Success | ❌ Failure / looping | | Object manipulation | ✅ Success | ✅ Success | ❌ More misoperations | | Multi-step planning | ✅ Success | ✅ Success | ❌ Dead ends |

When fidelity exceeds a threshold:

• Task success rates drop significantly
• LLMs show "attention distraction"—details pull them into goal-irrelevant actions
• Chain-of-Thought quality degrades—the model's inner monologue fills with unnecessary analysis of irrelevant details

Why Does This Happen?

The paper's explanation: an LLM's reasoning capacity is a finite "cognitive resource." When observations are too rich, the model spends a large token budget *describing the world* instead of *thinking about how to change the world*—like a child distracted in a candy store.

Mechanism: Cognitive Load Allocation

Analyzing internal attention patterns, the paper identifies two phenomena:

• Attention drift: At low fidelity, attention focuses on the task goal ("move the cup to the table"). At high fidelity, attention scatters across environmental minutiae ("the cup has 12 glints, a tiny edge chip, the table's wood grain runs at 37 degrees..."). This drift is not deliberate—it is the natural consequence of information overload.
• Hallucinated reasoning: Worse, high-fidelity data can trigger hallucination—the model invents constraints from irrelevant details. Seeing a subtle floor texture change, it may infer "this is a trap, I can't step there," when it's just ordinary flooring.

Deeper Implications: Occam's Razor Returns to Embodied AI

The significance goes beyond "give robots fewer sensors." It touches a fundamental principle:

> In intelligent systems, a "good enough" representation often beats a "maximally precise" one.

This aligns with the "functional representation" theory in cognitive science: humans navigating don't memorize the exact position of every stone—they build a topological map ("the post office is left of the supermarket, the park is right of the bridge"). This fuzzy-but-useful representation is precisely the basis of efficient decision-making.

The post also invokes a Feynman-style lesson: if an LLM can't complete a task on a "freshman-level" observation representation, it doesn't truly *understand* the task—it may just have been luckily misled into the right answer by some relevant clue buried in massive detail.

Why This Paper Matters

1. Counterintuitive yet widespread: Many researchers default to "more data = better"; this paper provides systematic counterevidence 2. Practical: It directly informs how to design robot sensor systems and data preprocessing pipelines 3. Fundamental: It reveals an LLM cognitive bottleneck—not insufficient data, but misallocated attention

Reference

Zenkri, O., & Brock, O. (2026). *When Higher Observation Fidelity Hurts Problem Solving*. arXiv preprint arXiv:2605.20072.

> *"Sometimes, covering one eye lets you see further."* 👁️