The Paper
| Property | Detail | |:---|:---| | Title | Probing Embodied LLMs: When Higher Observation Fidelity Hurts Problem Solving | | Authors | Oussama Zenkri, Oliver Brock | | Affiliation | Robotics and Biology Laboratory, Technische Universität Berlin | | arXiv ID | 2605.20072 | | Date | May 19, 2026 | | Category | cs.AI | | Core claim | Higher observation fidelity can degrade embodied LLM problem-solving; moderate noise improves performance by breaking repetitive action loops |
Imagine you walk into a room and face a locked metal chest with four movable levers. Your only way to open it is to flip the levers one by one — you don't know which lever connects to which, or in what order. You can only try.
Now consider three ways of playing:
1. Perfect transparency: after every lever flip, a display precisely shows the state of all levers. 2. An ordinary camera: you compare before/after photos to infer what changed. 3. Camera plus depth sensor: the photos gain 3D distance information.
Common sense says option 1 is fastest. It's wrong — the perfectly informed group was the slowest.
That is the counterintuitive result from Zenkri and Brock at TU Berlin. They built a physical Lockbox, had a real robot arm manipulate it, used GPT o1 as the brain, and provided three observation channels. The finding forces us to re-examine why we believe "more information is always better."
1. A Detective in a Lockbox
The Lockbox is a board with four two-position levers whose mechanical interdependencies (e.g., whether lever A can move depends on the positions of B and C) are completely hidden. This task has long been used in cognitive biology to study problem-solving in parrots, raccoons, and mice. It compresses the core challenge of real-world problem-solving: an action's effect depends on hidden state, and you must infer the dependency structure through interaction history.
The setup: a Franka Emika Panda arm with a soft gripper, an RGB-D camera, and a force sensor. At each step the robot picks a lever to flip; after each flip, a photo is taken and sent to GPT o1 for the next decision — a closed loop of see → think → act → see again.
2. Three Eyes, One Brain
- RGB: an ordinary color photo from a fixed viewpoint. The LLM must compare successive frames to determine which lever moved.
- RGB-D: adds an aligned depth map, theoretically providing 3D geometry.
- Ground-truth: a text output of exact lever states ("Joint A: position 0, Joint B: position 1..."). No visual reasoning needed — theoretically perfect.
- Loop incidence vs. success rate: strong negative correlation.
- Noise strength vs. loop incidence: non-monotonic, with the minimum at p≈40%.
- Only OpenAI models were tested (GPT o1 physical, GPT-4o simulation). DeepSeek, Claude, and Gemini are untested.
- Only one Lockbox instance physically. Simulation layout-transfer experiments preliminarily support generality, but real-world repetition is missing.
- The result cannot be directly generalized to all tasks. The Lockbox has a tiny state space (16 states), fixed dependencies, and cheap exploration. In high-stakes domains like autonomous driving, adding noise would likely be disastrous.
Each condition ran 10 independent trials with up to 20 manipulation steps.
3. The Counterintuitive Result
RGB performed best, ground-truth worst, RGB-D in between. RGB reached 80% success in 11 steps; ground-truth needed 15 steps for the same rate — a 36% step penalty for cleaner observations. Both plateaued at 80%.
Even stranger: the ground-truth group occasionally "hallucinated" state changes (~5% of steps) — impossible with perfect text input unless the model invented them. Meanwhile RGB had a 21.7% visual misjudgment rate, yet the less accurate group was more decision-efficient.
4. Dumping Noise into Perfect Observations
To investigate, the researchers ran simulation experiments with text-only state descriptions, randomly flipping the reported action outcome with probability p, from 0% to 60%, 30 trials per condition.
Success was not monotonically decreasing — it peaked at p=40%, reaching 2.85x the zero-noise success rate. The inverted-U curve went from 23.3% at 0% noise, rose to a peak at 40%, and fell again at 60%.
Spend an afternoon upgrading your AI's cameras and sensor precision to eliminate all uncertainty — and it turns out that letting its perception randomly lie 40% of the time nearly triples problem-solving efficiency.
5. The Broken Gyroscope: Repetitive Action Loops
The clue: LLMs persistently get stuck in loops — choosing A, B, C, then returning to A, B, C a few rounds later despite no change. The paper defines a repetitive action loop as an action subsequence of length ≥3 appearing at least twice in a trial.
Two correlations emerged:
When feedback is perfectly consistent — "you flipped A, and A moved" — the model tends to repeat the same exploration pattern, because every round confirms its prior assumptions. Random erroneous feedback interrupts this: the model must stop and reconsider why its expectation failed. As the paper puts it: "Accurate feedback may sustain repetitive action loops, whereas erroneous observations can disrupt them."
Some degree of "perceptual failure," in other words, acts as a necessary condition for decision flexibility.
6. What This Means — and What It Doesn't
Honest caveats:
In the real world, perception is always noisy, reasoning always imperfect, environments always changing. If you evaluate your AI only under pristine conditions — clean rooms, fixed cameras, perfect annotations — are you measuring your AI's ability, or the artificiality of your test environment?
7. A Larger Lesson
Physics has a concept called "noise-induced order": in some nonlinear systems, small random perturbations help escape local optima and find better global states. Simulated annealing works on exactly this principle.
This paper's pattern is reminiscent. LLM reasoning loops resemble a system trapped at a local energy minimum. Perfect feedback prevents it from "seeing" alternatives — every step returns the expected answer, so it keeps walking the same path. Randomly flipped observations inject thermal motion, occasionally forcing deviations that uncover genuinely useful action sequences.
But don't misread this as "noise is good." Noise merely rescued the model from a worse failure mode under this particular task structure. The real solution is making models avoid loops without noise — that's the hard part. Viewed this way, the paper is a mirror exposing a structural weakness of LLMs in closed-loop embodied reasoning: they lack an intrinsic exploration drive. When observations say everything is fine, their default is repetition, not variation. Animals have curiosity; current LLMs do not.
Conclusion
The authors write: "In embodied scenarios where perception and reasoning are simultaneously imperfect and hard to disentangle, success rates may reflect the incidental interplay of these imperfections rather than genuine problem-solving ability."
That sentence deserves to be pinned above every robot-plus-LLM researcher's monitor. We tend to treat observation as a passive input to decisions — clearer input leads to better decisions, linear thinking. This paper, with a physical lockbox and a series of elegant experiments, shows that observation actively shapes decision dynamics. Give the model different eyes, and it becomes a different thinker.
Sometimes "knowing less" makes you more flexible. That isn't anti-intellectual — it's a reminder not to mistake measured success for true capability.