MineExplorer: 18 Models Tested in Minecraft — Top Model Drops from 77.69 on Single-Hop Tasks to 12.34 on Four-Hop

It's getting dark. You spawn in an unfamiliar forest, you're hungry, and a spider is crawling toward you.

Send a screenshot to a multimodal model, and it will mostly know "there's a monster, danger." The problem is, the world has no pause button. Over the next three minutes, the model must move on its own, gather resources, craft tools, avoid attacks, and figure out prerequisite steps that the instruction never spelled out.

Meituan LongCat and Shanghai Jiao Tong University turned this scenario into MineExplorer. The paper went up on arXiv in late May, and on July 23 the team released a full write-up along with consolidated links to the code, dataset, and a runnable environment. The benchmark deliberately filters out tasks that rely heavily on the Minecraft wiki and does not test Minecraft-specific trivia—it uses a controllable 3D sandbox to measure general-purpose open-world exploration. [1][2]

How the Exam Is Built

MineExplorer retains 813 human-verified composite instances, covering 1-hop through 4-hop tasks. Each task runs for up to 1,800 environment steps at 0.1 seconds per step—exactly three minutes of continuous interaction. The final goal is given to the model, but the dependency graph is not laid out: to "obtain some item," the model may first need to recognize terrain, find raw materials, craft tools, and then take a detour to get there.

The evaluation breaks capability into 14 metrics grouped into perception, reasoning, and action; milestones are checked by rules, not scored offhandedly by a large-model judge. Tasks are generated collaboratively by five agents and then curated by humans. The code repository provides generation scripts, evaluation scripts, Docker images, and an HTTP-API-based Minecraft sandbox, all under the MIT license. It is both an exam and a factory for making new questions. [2][3]

The Scores Are Ugly. That's Why They're Useful

The team tested 18 models. The best, Claude-Opus-4.6, achieved an overall task success rate (TSR) of only 41.08. The breakdown is starker: 77.69 on 1-hop, 32.68 on 2-hop, 20.69 on 3-hop, and just 12.34 on 4-hop. Every additional layer of hidden prerequisites cuts the success rate down a notch. [1]

It's not that the model can't see. Claude-Opus-4.6 scored 61.91 on perception and 54.71 on reasoning; in failure attribution, navigation accounts for nearly 60%. The model can describe what's in front of it, but it frequently fails to align past observations, current position, and the path to the goal. On-paper "understanding of the world" leaks away in the closed loop.

Two ablations are even harsher:

• More time doesn't help. Tasks that can be completed are usually finished early; those still unsolved at 1,800 steps mostly just wander in circles if given more.
• More memory doesn't help either. Increasing history frames from 1 to 20 raised TSR from 23.00 to 41.08; pushing to 50 frames dropped it back to 37.64. Old frames become noise that interferes with current judgment.

This mirrors AI coding: stuffing more repository context or a larger token budget into the prompt does not automatically raise task completion. What matters is state compression, goal dependencies, error recovery, and "what has the world actually become right now."

Why "Soft Embodiment"

MineExplorer has a perception–reasoning–action loop and a 3D world that changes with actions, so it is genuinely closer to an embodied agent than static VQA. But it remains simulated embodiment: the action interface is discrete, physics is deterministic, and there are no motor errors, touch, sensor drift, continuous control, or real safety costs. Being able to navigate Minecraft does not mean a robot can enter a factory.

Its value lies precisely here. Trial and error with real robots is expensive, slow, and dangerous; Minecraft is cheap, reproducible, and can mass-generate hidden-prerequisite tasks. First test long-horizon planning, state alignment, and recovery thoroughly in soft-embodied environments—then talk about real hardware. Getting the order right saves a lot of money.

What MineExplorer offers is not the emotional conclusion that "large models don't work." It pinpoints a specific crack: between seeing the world and consistently accomplishing goals in a world that changes lies the long-horizon closed loop. Today's strongest model nearly passes on single-hop; on four-hop, it still walks like a toddler.

Sources

1. MineExplorer paper (arXiv:2605.30931) 2. Meituan LongCat team full write-up 3. MineExplorer GitHub repository · Hugging Face dataset