MineExplorer: 18 Multimodal Models Tested in Minecraft — Best Model Drops from 77.69 on Single-Hop to 12.34 on Four-Hop Tasks

Forum topic · 小凯 · 2026-07-26

Summary

MineExplorer is an open-world agent benchmark built by Meituan LongCat and Shanghai Jiao Tong University, released on arXiv in late May and with a full technical write-up plus code, datasets, and a runnable Minecraft sandbox environment published on July 23. The benchmark keeps 813 human-verified composite tasks spanning 1-hop to 4-hop dependency chains, each limited to 1,800 environment steps (about three minutes of continuous interaction). Capabilities are split into 14 items across perception, reasoning, and action, with milestones checked by rules rather than LLM judges. Results across 18 tested models are sobering: the best model, Claude-Opus-4.6, achieves only 41.08 overall task success rate (TSR), falling from 77.69 on 1-hop tasks to 32.68, 20.69, and just 12.34 as hidden prerequisites stack to 2-, 3-, and 4-hop. Ablations show that neither more time nor more memory helps—failure is dominated by navigation (nearly 60%), revealing a gap between seeing the world and sustaining goal-directed action in a changing environment.

MineExplorer: 18 Models Tested in Minecraft — The Strongest Drops from 77.69 on Single-Hop to 12.34 on Four-Hop Tasks

It's getting dark. You've spawned in an unfamiliar forest, you're hungry, and a spider is crawling toward you.

Show a screenshot to a multimodal model, and most will know "there's a monster, danger." The problem is, the world has no pause button. Over the next three minutes, the model has to move on its own, gather resources, craft tools, avoid attacks, and figure out the prerequisite steps the instruction never spelled out.

Meituan LongCat and Shanghai Jiao Tong University turned this into MineExplorer. The paper went up on arXiv in late May; on July 23 the team released a full technical walkthrough, consolidating code, datasets, and entry points to a runnable environment. It deliberately filters out tasks that lean heavily on the Minecraft Wiki — it doesn't test Minecraft trivia — and instead uses a controllable 3D sandbox to measure general open-world exploration. [1][2]

How the Exam Is Built

MineExplorer retains 813 human-verified composite instances, spanning 1-hop to 4-hop dependency depth. Each task runs for at most 1,800 environment steps at 0.1 seconds per step — exactly three minutes of continuous interaction. The final goal is given to the model, but the dependency graph is not laid out: to "obtain some item," the model may first need to identify terrain, find raw materials, craft tools, and then take a detour to get there.

The evaluation splits capabilities into 14 items grouped under perception, reasoning, and action; milestones are verified by rules, not scored on the fly by an LLM judge. Tasks themselves are generated by five collaborating agents and then curated by humans. The code repository provides generation scripts, evaluation scripts, Docker images, and an HTTP-API-based Minecraft sandbox, under the MIT license. It is both an exam paper and a factory for producing more questions. [2][3]

The Scores Are Ugly. And That's Why They're Useful

The team tested 18 models. The best, Claude-Opus-4.6, achieves an overall task success rate (TSR) of only 41.08. Breaking it down is even more striking: 77.69 on 1-hop, 32.68 on 2-hop, 20.69 on 3-hop, and just 12.34 on 4-hop. Every additional hidden prerequisite layer cuts success rate sharply. [1]

It's not a perception problem. Claude-Opus-4.6 scores 61.91 on perception and 54.71 on reasoning; in failure attribution, navigation accounts for nearly 60%. The model can describe what it sees, but often fails to align past observations, current position, and the path to the goal. "Understanding the world" on paper leaks all over the place once put into a closed loop.

Two ablations are even harsher:

More time doesn't help. Tasks that can be completed are usually done early; those still unsolved at 1,800 steps mostly keep wandering in circles.
More memory doesn't help. Increasing history frames from 1 to 20 raises TSR from 23.00 to 41.08; pushing further to 50 frames drops it back to 37.64. Old frames become noise that interferes with current judgment.

This echoes AI coding. Stuffing more repository context or granting a larger token budget does not automatically raise task completion. What matters is state compression, goal dependencies, error recovery, and "what has the world actually become right now."

Why It's Called "Soft Embodiment"

MineExplorer has a perception–reasoning–action closed loop and a 3D world that changes with actions — genuinely closer to an embodied agent than static VQA. But it is still simulated embodiment: the action interface is discrete, physics rules are stable, and there are no motor errors, touch, sensor drift, continuous control, or real safety costs. Being able to navigate in Minecraft does not mean a robot can enter a factory.

Its value lies precisely here. Trial and error on real robots is expensive, slow, and dangerous; Minecraft is cheap, reproducible, and can mass-generate hidden-prerequisite tasks. Test long-horizon planning, state alignment, and recovery in soft-embodied environments first, then talk about real hardware. Getting the order right saves a lot of money.

MineExplorer's conclusion is not the emotional "large models don't work." It pins down a specific crack: between seeing the world and persistently completing goals in a world that changes lies the long-horizon closed loop. Today's strongest models nearly pass single-hop; on four-hop, they still look like toddlers just learning to walk.

References

1. MineExplorer paper (arXiv:2605.30931) 2. Meituan LongCat team full walkthrough 3. MineExplorer GitHub repository · Hugging Face dataset