MineExplorer: 18 AI Models Tested in Minecraft — Best Model Falls from 77.69 on 1-Hop Tasks to 12.34 on 4-Hop

It's getting dark. You spawn in an unfamiliar forest, you're hungry, and a spider is crawling toward you.

Show a screenshot to a multimodal model and it will usually know "there's a monster, it's dangerous." The problem is the world has no pause button. Over the next three minutes, the model must move on its own, gather resources, craft tools, avoid attacks, and infer prerequisite steps that were never spelled out in the instruction.

Meituan LongCat and Shanghai Jiao Tong University turned this scenario into MineExplorer. The paper went up on arXiv in late May, and on July 23 the team published a full technical explainer, along with consolidated links to the code, datasets, and a runnable environment. The benchmark deliberately filters out tasks that depend heavily on the Minecraft Wiki — it doesn't test Minecraft trivia, but uses a controllable 3D sandbox to probe general open-world exploration. [1][2]

How the Testing Ground Is Built

MineExplorer keeps 813 human-verified composite instances, covering 1-hop through 4-hop tasks. Each task runs for up to 1,800 environment steps at 0.1 seconds per step — exactly three minutes of continuous interaction. The final goal is given to the model, but the dependency graph is not laid out: to "obtain some item," the model may first need to recognize terrain, find raw materials, craft tools, and then take a detour to get there.

The evaluation decomposes capability into 14 items across three groups — perception, reasoning, and action. Milestones are verified by rules rather than scored loosely by a large-model judge. Tasks themselves are generated by five collaborating agents and then hand-curated. The code repository includes generation scripts, evaluation scripts, Docker images, and an HTTP-API-based Minecraft sandbox, under the MIT license. It serves both as an exam paper and as a factory for making new questions. [2][3]

The Scores Are Ugly. That's Why They're Useful.

The team tested 18 models. The best, Claude-Opus-4.6, achieved an overall task success rate (TSR) of only 41.08. The breakdown is more striking: 77.69 on 1-hop, 32.68 on 2-hop, 20.69 on 3-hop, and just 12.34 on 4-hop. Every additional layer of hidden prerequisites cuts the success rate sharply. [1]

It's not that the model can't see. Claude-Opus-4.6 scored 61.91 on perception and 54.71 on reasoning; in failure attribution, navigation accounted for nearly 60%. The model can describe what's in front of it, but frequently fails to align past observations, current position, and the path toward the goal. "Understanding the world" on paper leaks badly once it's inside a closed loop.

Two ablation experiments are even harsher:

• More time doesn't help. Tasks that can be solved are usually solved early; among tasks still unsolved at 1,800 steps, letting them run longer mostly just means wandering in circles.
• More memory doesn't help. As history frames increased from 1 to 20, TSR rose from 23.00 to 41.08; but pushing to 50 frames dropped it back to 37.64. Old frames became noise interfering with current decisions.

This mirrors AI coding. Stuffing more repository context into the prompt or granting a bigger token budget does not automatically raise task completion. What matters is state compression, goal dependencies, error recovery, and knowing "what the world actually looks like right now."

Why "Soft Embodiment"

MineExplorer has a perception–reasoning–action closed loop and a 3D world that changes with actions — genuinely closer to an embodied agent than static VQA. But it is still simulated embodiment: the action interface is discrete, physics rules are stable, and there are no motor errors, tactile feedback, sensor drift, continuous control, or real safety costs. Being able to navigate in Minecraft doesn't mean a robot can enter a factory.

Its value lies precisely here. Trial and error with real robots is expensive, slow, and dangerous; Minecraft is cheap, reproducible, and can batch-generate tasks with hidden prerequisites. Test long-horizon planning, state alignment, and recovery in soft-embodied environments first — then talk about real hardware. Getting the order right saves a lot of money.

What MineExplorer offers is not the emotional conclusion that "large models don't work." It pinpoints a specific crack: between seeing the world and persistently completing goals in a world that changes, there lies a long-horizon closed loop. Today's strongest models nearly pass the 1-hop test; on 4-hop, they still seem to be learning to walk.

Sources

1. MineExplorer paper (arXiv:2605.30931) 2. Meituan LongCat team's full explainer 3. MineExplorer GitHub repository · Hugging Face dataset