Lost in the Perfect Map: LLMs Encode Structure but Fail to Use It In-Context

The Paradox: A Perfect Map You Cannot Walk

Imagine holding an exquisitely precise city map—you can reconstruct every street in your mind, yet you cannot take a single step from the café to the bookstore. According to a 2025 paper by Google DeepMind researchers with Brown University and NYU, *Language Models Struggle to Use Representations Learned In-Context* (arXiv:2602.04212), this paradox describes large language models (LLMs): they can build near-perfect internal "maps" of a world but often cannot use those maps to act.

Background: In-Context Learning

In-context learning (ICL) is one of LLMs' signature abilities—given a few examples in the prompt, with no parameter updates, the model appears to instantly learn new tasks (e.g., translating 苹果 → apple, then correctly answering 西瓜 → watermelon). Built on the Transformer's self-attention mechanism since 2017, ICL underpins everything from chatbots to code completion, fueling hopes of broad, general capability.

The Experiments

The researchers placed models in minimal virtual worlds defined by topology:

• 1D linear chains (like a 16-stop subway line)
• 2D grids (4×4 or 5×5)

Prompts described the world's structure plus navigation examples; models then had to predict multi-step path outcomes.

Results

| Model | Params | Topology | Steps | Accuracy | Representation quality (distance correlation) | |-------|--------|----------|-------|----------|------------------------------------------------| | Gemma 3 | 4B | 1D chain | 1 | ~95% | ~90% | | Gemma 3 | 4B | 2D grid (4×4) | 2 | <20% | ~85% | | Gemma 3 | 27B | 1D chain | 3 | ~60% | ~92% | | Gemma 3 | 27B | 2D grid (5×5) | 3 | <15% | ~85% | | GPT-5 | undisclosed | 2D grid (5×5) | 2 | ~25% | undisclosed |

The Eerie Finding: The Map Is Perfect

Analyzing internal states, the researchers found the models encode the world's geometry almost flawlessly:

1. Distance correlation (~85% even on 2D grids): the model internally reconstructs distances that closely match the true topology. 2. Dirichlet energy (low values): the representations are smooth and well-structured.

The knowledge is encoded—but cannot be used. The authors call this "inert knowledge": like a navigation app with flawless map data that never moves when you press "Start."

The Culprit: The Transformer's One-Dimensional Soul

The researchers point to self-attention, which is inherently built for 1D sequences. Text is linear; 2D structures require simultaneously tracking multiple independent directions—a kind of dimensional curse for attention.

Rescue attempts largely failed:

• Extending context to 5000 tokens
• Chain-of-thought prompting
• Stronger closed-source models

Accuracy barely moved—suggesting the problem is not perception but the inability to *use* what is perceived.

Why It Matters

• Everyday use: models write elegant poetry yet fail multi-step logic; they summarize literature yet stumble when simulating physical processes.
• High-stakes domains: in autonomous driving, medical diagnosis, finance, or robotics, "knowing but not acting" could turn a bug into a catastrophe.
• Related work: DeepMind's separate "knowing-doing gap" research showed LLMs form accurate reasoning chains but fail to convert them into action; reinforcement learning fine-tuning (RLFT) raised action coverage from 40% to 52% and tic-tac-toe win rate from 15% to 75%—evidence the gap can be narrowed with new training methods.

The Deeper Debate: Imitation vs. Understanding

• Imitation view: LLMs are statistical approximators optimized to generate artifacts, not adaptive agents—they can "know" structure without reliably "doing" tasks.
• Understanding view: some researchers argue certain LLM capabilities reflect aspects of human cognition, and "artificial neuroscience" reveals functional similarities between models and brains.

As Google's Blaise Agüera y Arcas and James Manyika write in "AI Is Evolving — And Changing Our Understanding Of Intelligence", dramatic AI advances compel us to rethink what intelligence truly is—a potential paradigm shift in how we define "understanding."

Paths Forward

1. Reinforcement learning fine-tuning (RLFT) — bridge reasoning and action via environment interaction and reward signals. 2. Hybrid architectures — offload complex computation to external tools (calculators, map APIs, simulators). 3. Neurosymbolic AI — combine neural pattern recognition with symbolic logical reasoning. 4. Non-Transformer architectures — designs that natively handle multidimensional structure rather than simulating it on 1D sequences.

Conclusion

LLMs are like a brilliant young scholar who reads voraciously and draws beautiful knowledge maps but stumbles when actually stepping outside. True understanding requires not just *knowing* but *using*. Until AI can walk freely across its own maps, a measure of clear-eyed skepticism—and manual verification at critical moments—remains wise.

References

1. Lepori, M. A., et al. (2025). *Language Models Struggle to Use Representations Learned In-Context*. arXiv:2602.04212. https://arxiv.org/abs/2602.04212 2. Park, C. F., et al. (2025). *ICLR: In-Context Learning of Representations*. ICLR 2025. 3. DeepMind (2025). *LLMs Struggle to Act on What They Know: Bridging the Knowing-Doing Gap via Reinforcement Learning Fine-Tuning*. https://www.marktechpost.com/2025/05/18/llms-struggle-to-act-on-what-they-know/ 4. Agüera y Arcas, B. & Manyika, J. (2025). *AI Is Evolving — And Changing Our Understanding Of Intelligence*. Noema Magazine. https://www.noemamag.com/ai-is-evolving-and-changing-our-understanding-of-intelligence 5. Subasioglu, M. & Subasioglu, N. (2025). *From Mimicry to True Intelligence (TI): A New Paradigm for Artificial General Intelligence*. arXiv:2509.14474. 6. Lâasri, H. (2025). *Deep Research in AI: The Insight Gap*. https://medium.com/data-science-collective/deep-research-in-ai-the-insight-gap-446118ebe76e