MineExplorer: 18 Multimodal Models Tested in Minecraft—Best Model Falls from 77.69 to 12.34 as Task Complexity Rises
It's getting dark. You spawn in an unfamiliar forest, you're hungry, and a spider is crawling toward you.
Screenshot that scene for a multimodal model and it will usually know: "there's a monster, danger." The problem is, the world has no pause button. Over the next three minutes, the model must move on its own, gather resources, craft tools, dodge attacks, and infer prerequisite steps that were never spelled out in the instruction.
Meituan LongCat and Shanghai Jiao Tong University turned this into MineExplorer. The paper went up on arXiv in late May, and on July 23 the team published a full technical write-up, centralizing links to the code, datasets, and a runnable environment. The benchmark deliberately filters out tasks that depend heavily on the Minecraft Wiki—it doesn't test Minecraft trivia, but uses a controllable 3D sandbox to measure general open-world exploration. [1][2]
How the testbed is built
MineExplorer retains 813 human-verified composite instances, covering 1-hop through 4-hop tasks. Each task runs for at most 1,800 environment steps at 0.1 seconds per step—exactly three minutes of continuous interaction. The final goal is given to the model, but the dependency graph is never laid out: to "pick up item X," the model may first need to recognize terrain, find raw materials, craft tools, and then navigate around obstacles.
The evaluation decomposes capability into 14 metrics across perception, reasoning, and action; milestones are checked by rules, not scored by an LLM judge. Tasks themselves are generated by five cooperating agents, then human-filtered. The repository provides generation scripts, evaluation scripts, a Docker image, and an HTTP-API-based Minecraft sandbox, all under MIT license. It serves as both an exam and a question factory.
The scores are ugly—and that's what makes them useful
The team tested 18 models. The best, Claude-Opus-4.6, achieved an overall task success rate (TSR) of only 41.08. The breakdown is starker: 77.69 on single-hop, 32.68 on 2-hop, 20.69 on 3-hop, and just 12.34 on 4-hop. Every added hidden prerequisite layer cuts the success rate sharply. [1]
It's not a perception problem. Claude-Opus-4.6 scored 61.91 on perception and 54.71 on reasoning; in failure attribution, navigation accounts for nearly 60%. The model can describe what's in front of it, yet often fails to align past observations, current position, and the goal path. "Understanding the world" on paper leaks badly once placed in a closed loop.
Two ablations are even harsher:
- More time doesn't help. Tasks that can be solved are usually solved early; among tasks still unsolved at 1,800 steps, most just keep wandering.
- More memory doesn't help either. Raising history frames from 1 to 20 lifted TSR from 23.00 to 41.08; pushing to 50 frames dropped it back to 37.64. Old frames become noise that interferes with current decisions.
Why "soft embodiment"
MineExplorer has a perception–reasoning–action loop and a 3D world that changes with actions, so it is genuinely closer to an embodied agent than static VQA. But it remains *simulated* embodiment: discrete action interfaces, stable physics, no motor error, no touch, no sensor drift, no continuous control, no real safety costs. Navigating Minecraft doesn't equal getting a robot into a factory.
Its value lies precisely here. Real-robot trial and error is expensive, slow, and dangerous; Minecraft is cheap, reproducible, and can mass-generate hidden-prerequisite tasks. Test long-horizon planning, state alignment, and recovery in soft-embodied environments first—then talk about real hardware. Getting the order right saves a lot of money.
MineExplorer's takeaway is not the emotional conclusion that "large models don't work." It pins down a specific crack: between seeing the world and sustainably completing goals in a world that changes lies the long-horizon closed loop. Today's strongest models nearly pass single-hop; on four-hop, they still walk like toddlers.
References
1. MineExplorer paper (arXiv:2605.30931) 2. Meituan LongCat team's full write-up 3. MineExplorer GitHub repo · Hugging Face dataset