Letting Machines Dream — From "Only Dreaming of Places I've Been" to "Dreaming of Places I've Never Been"

Imagine you're learning to ride a bicycle. You circle a plaza for an hour; every right turn goes smoothly, but you fell three times on left turns. After falling, you keep thinking it over: was my weight too far right during the turn? Was the pedal at the wrong angle? You replay that left-turn scene in your head many times, imagining different counterfactual responses.

Now here's a question: have you ever imagined "riding a bike somewhere you've never been"? For instance—imagining yourself taking a sharp downhill switchback on a mountain road.

Probably not. Because your imagination's "ignition" requires an experienced scenario first. All your counterfactual reasoning is based on visual scenes and bodily sensations you've actually lived through. That's a limitation—your imagination is "tethered" by your experience.

Xu, Zhou, Lin, Meng, Ji, Shi, and Zhao recently posted a paper on arXiv (2605.16030) addressing exactly this problem—not for humans, but for robots.

⛓️ The Tether of Imagination

Model-based reinforcement learning (MBRL) is one of the core methods for teaching robots complex tasks. The idea: let the robot build a model of the world inside its "mind"—a simulator that can predict "if I move the left wheel, what happens?"—and then do "imaginative training" inside that model. You don't need a real robot to fall a thousand times in the real world; you let it fall a thousand times in its imagination, and transfer the learned experience to reality.

That sounds clever. But it has a fatal problem—called "Historical Tethering" in the paper's abstract.

What does it mean? MBRL's "imagination" always begins at an actually observed state. The robot picks a real state from its experiential memory and imagines the future starting from there. It simply cannot start imagining from a state it has never reached.

This creates an asymmetry: the robot's world model keeps expanding—each step of real-world exploration teaches it a bit more—but the policy stays stuck in a narrow region. The policy can't explore optimal paths from unseen starting points, because it can't "dream" from those starting points.

You could put a robot in a complex initial state it has never seen—say, a humanoid robot on a steeply tilted slope—but it cannot rehearse this scenario in advance, because it has no record of "starting from the slope."

💭 Mind Dreamer: Untethering Imagination

This paper's contribution is direct: they give the robot a "dream maker"—a generative model that doesn't sample starting states from historical experience, but synthesizes new starting states on its own.

These synthetic states aren't random. They must satisfy two conditions: physically plausible (it can't generate a head-down robot standing on the ceiling), yet cognitively challenging (precisely the situations the robot's policy currently handles worst).

How do they achieve "precisely challenging"? The generator is trained adversarially—generator and policy play against each other. The generator tries to produce states the policy can't handle; the policy tries to learn to handle them. Like two chess players: one racking their brain for tricky moves, the other for counters.

🌀 Jumping Across Discontinuities

When imagination starts from an observed state, the imagined trajectory is continuous—from A to A+1 to A+2, every step predictable. But when imagination starts from a generated state, there's a "gap" between all experienced states and this new one—because your world model has never seen this starting point.

How do you price this gap? How do you know "the value of starting from a synthetic, never-actually-visited state"?

That's what their Relay Value Function and Relay Uncertainty Function do. They treat the synthesized starting state as a "counterfactual intermediate state"—even though the robot has never been there, the value and uncertainty functions can propagate "is it worth going?" information across the gap.

There's a theoretical result I found especially interesting: they prove that when propagating uncertainty across such discontinuous jumps, the discount factor must be squared—γ² instead of γ. Intuitively: ordinary timesteps carry one uncertainty factor (γ), but crossing a synthetic gap requires two—one for "I don't know this state itself," and one for "I don't know the future from this state." This γ² has no counterpart in standard reinforcement learning.

> Honestly, I can't fully grasp the γ² derivation from the abstract alone. It relies on their formalized framework of "uncertainty propagation" and a Bellman-style recursive structure I'm familiar with, but the concrete derivation steps—especially why exactly γ² rather than some nonlinear decay—are not spelled out in what I could see. My intuition says the conclusion is reasonable, but claiming I fully understood the proof would be lying to myself.

📊 How Well Does It Work?

They ran experiments on the DeepMind Control Suite. Compared with the current strongest baseline, DreamerV3, Mind Dreamer is on average 1.67× faster on benchmark tasks. On the trickiest sparse-reward tasks—where reward signals are so rare that traditional methods barely learn at all—the speedup reaches 8.8×.

In other words, when a robot starts imagining from states it "dreamed" rather than "experienced," it has already rehearsed in its head how to handle difficulties before actually encountering them.

🤷 Things I Didn't Figure Out

At least three open questions.

First, how does the generator guarantee synthetic states are "physically plausible"? The abstract only says it uses adversarial training. But "physical plausibility" is a vague condition—in a simulator, spatial plausibility might be relatively easy to judge, but plausibility involving physical laws (gravity, friction, contact forces) is far more complex. I don't know how they did it specifically, nor whether the method transfers from simulation to real robots.

Second, the computational cost of the Relay Value Function and Relay Uncertainty Function. Every "dream" requires evaluating both functions, and these may depend on some global estimate over the entire state space. That might be acceptable in simulation, but on a real robot, compute could be the bottleneck.

Third, the "quadratic discount" (γ²) result makes me curious but also uncertain. How universal is it? Which conditions of their Relay Manifold assumption does it depend on? If the manifold's topology is more complex (with holes, with crossings), does γ² still hold? I can't judge.

🌙 But the Core Idea Is Hard to Refute

If you can't start imagining from places you've never been, you can't learn. This paper offers a very direct, very engineering-minded way to break that limitation: build a generator to "manufacture difficulty," then let the policy learn to solve it. "Dreaming" isn't some mystical term—it's just a special form of training data augmentation, except what's augmented isn't the input but the starting point of imagination.

---

References

1. Xu, S., Zhou, X., Lin, Y., Meng, Y., Ji, X., Shi, L., & Zhao, R. (2026). *Mind Dreamer: Untethering Imagination via Active Latent Intervention on Latent Manifolds*. arXiv:2605.16030 [cs.LG]. https://arxiv.org/abs/2605.16030 2. Hafner, D., et al. (2023). *DreamerV3: Mastering Diverse Domains through World Models*. arXiv:2301.04104. 3. Ha, D., & Schmidhuber, J. (2018). *World Models*. arXiv:1803.10122. 4. Sutton, R. S. (1991). *Dyna, an Integrated Architecture for Learning, Planning, and Reacting*. ACM SIGART Bulletin, 2(4), 160-163. 5. Friston, K., et al. (2017). *Active Inference: A Process Theory*. Neural Computation, 29(1), 1-49.