From Discovery to Building: The AI Creativity Gap Revealed by Minecraft
Paper: Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft Authors: Zhou Ziheng, Huacong Tang, Jinyuan Zhang, Haowei Lin, Bangcheng Yang, Qian Long, Fang Sun, Yizhou Sun, Yitao Liang, Ying Nian Wu, Demetri Terzopoulos, Xiaofeng Gao arXiv: 2604.24697
Why Minecraft?
There is a large complexity gap between scientific discovery and real-world engineering: discovering a physical law is far easier than building a working system that exploits it. Existing AI evaluations are either too easy (models pass by memorization) or too hard to attribute failures (real robotics). Minecraft sits at a sweet spot, and its redstone system is a perfect micro-engineering universe:
- Simple physical rules (signal propagation, logic gates, timing delays)
- Combinatorial explosion (a few basic components can build CPUs, calculators, automatic farms)
- Clear success criteria (the lamp is on or off)
- A closed environment that excludes real-world noise
- Varying numbers of lamps
- Varying lighting patterns (simultaneous, sequential, specific timing)
- Varying available space, distance, and delay constraints
- Knowledge application remains the biggest weakness for all models — converting abstract rules into concrete builds requires spatial reasoning, temporal reasoning, and resource management.
- Knowledge gap identification is emerging as an equally severe obstacle for frontier models. The new failure mode is not 'doing it wrong' but 'not knowing what you did wrong': models blindly proceed, unaware that the task contains structures they have never seen.
- For researchers: new architectures must cultivate 'problem awareness,' not just execution ability.
- For industry: don't overestimate current AI's autonomous construction capability; human engineers remain irreplaceable where creative application of principles is required.
- For AI safety: a system that 'doesn't know what it doesn't know' poses far greater open-world risk than one that knows its limits.
- Zhou Z. et al. (2026). *Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft*. arXiv:2604.24697.
- Yao S. et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. *ICLR*.
- Xu F.F. et al. (2025). TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. *NeurIPS*.
Redstone construction requires a genuine discovery-to-application loop: parameterized tasks mean every level has different inputs, so agents must understand the underlying causal rules rather than recite circuit diagrams.
The SciCrafter Benchmark
The core task: build redstone circuits so that a set of lamps light up in a specified pattern. Parameterized design ensures each attempt demands real discovery:
Complexity and required knowledge grow non-linearly as target parameters scale up.
The authors dissect the discovery-to-application loop into four sub-skills:
1. Knowledge Gap Identification — knowing what you don't know 2. Experimental Discovery — finding unknown rules via trial and error 3. Knowledge Consolidation — organizing scattered findings into usable knowledge 4. Knowledge Application — turning knowledge into concrete building plans
The Shocking 26% Ceiling
Frontier models tested — GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 — all plateaued at roughly 26% success rate on SciCrafter. More than three-quarters of attempts by the most powerful models fail at 'understand the principle, then build.'
To diagnose where the 74% failure rate comes from, the authors designed targeted interventions, each aimed at one of the four sub-skills, using the marginal contribution of each intervention to identify the bottleneck — like testing four organs to find the diseased one.
The Shifting Bottleneck: From 'How' to 'What'
Two key findings:
The paper's precise verdict:
> "The bottleneck is shifting from 'solving problems correctly' to 'asking the right questions.'"
This points to a deep lack of metacognition: models can answer 'what is X' but struggle to autonomously generate questions like 'does X exist?'
SciCrafter as a Diagnostic Probe
The authors position SciCrafter as a diagnostic probe, not a final benchmark — like an ECG that locates a problem rather than cures it. Implications:
Feynman's insight applies: naming is not understanding. Real understanding must include operability — SciCrafter quantifies the famous competence-performance gap as a 74% failure rate.
Closing Thought
Today's AI is not Icarus — it has no wings yet. It is a flight student who has memorized the manual but never left the ground: it can explain thermals, draw perfect wing cross-sections, and land a hundred times in a simulator. But hand it a box of parts and say 'build a flying machine,' and the success rate is 26%.
That is not an insult — it is clarity. The distance between 'understanding' and 'creating' is not zero, but it is not infinite either. 26% means possibility exists; we just haven't crossed some undiscovered cognitive threshold yet.