99% on Trap Questions, Defeated by an Unwashed Knife: Where Judgment Models Hit a Hard Boundary

*English translation/summary of a zhichai.net forum post reviewing arXiv:2610.01834, part 2 of the "Three Layers of the Capability Map" series.*

Key points

• The model: Jev (jev-1.13.0, TypeSafe's System One series) is a judgment model: given a JSON state plus candidate options, it returns probabilities, a choice, and a confidence — no reasoning text, no chain of thought. All "thinking" happens inside probability assignment.
• Evaluation vs simulation: A decision is *evaluation* when the correct option is decidable from the input (verify each option against stated conditions). It requires *simulation* when correctness depends on predicting facts not described in the input (opponent moves, sub-goals, physical consequences). Crucially, this is a property of the task presentation, not the model: rewriting options to include consequences collapses simulation back into evaluation.

Four experimental arenas

1. Cognitive Reflection Test: On 150 novel CRT items (bat-and-ball style traps), Jev answers 99% correctly (humans fall for intuition 55% of the time; old GPT-3 80%). Pure evaluation is essentially solved. 2. Matrix games: 210 iterated-dominance-solvable 3×3 games, depths D=0–3, conditions plain/reason/told. With a dominant action (pure evaluation): perfect. Aligned games (a level-1 surface shortcut happens to be correct): 77–93%. Misleading games (shortcut points to a lure): 28–37%, with 53–71% of errors landing exactly on the lure. Asking it to "reason first" (reason condition) doesn't help; telling it the opponent's equilibrium action (told) jumps to 97–99%. Its choices are best fit (KL 0.068, 77%) by a quantal level-1 model — it scores actions by expected payoff assuming a random opponent. 3. ALFWorld (textual kitchen): On expert-trajectory steps, evaluation steps score 93–96%; aligned simulation-requiring (S-known) steps 95–99%; misleading S-known steps collapse to 36–43%, choosing the lure at ~5× random baseline. Example: task "put a clean butterknife in drawer" while holding a dirty knife — it picks "go to drawer 1" (0.60) over "go to sinkbasin 1" (0.17), then immediately picks the same drawer command once the knife is clean. It isn't confused about the drawer; it can't tell *when* to go there. 4. Robotic Stacker (DM Control): Driving raw joint torques every 0.02 s — consequences are pure physics not in the input — Jev succeeds 0/20, same as random. Given a code-executed skill library (move, descend, grasp, place…), it ties hand-written controllers on the trained task (9 vs 8) and crushes them on five unseen instructions (22/40 vs 3/40).

Probing: knowledge gap or architectural limit?

• Asked separately in its own call, Jev predicts the opponent's action correctly 90% of the time; given the opponent's action, it best-responds 98%.
• Yet among plain-condition calls where its own prediction was right, it chose correctly only 32% (35% when its prediction was wrong) — its choice is statistically independent of its own prediction.
• Two-call fix: code asks for a prediction, writes it back into the state, then asks for the choice. Misleading-game accuracy rises 33% → 94%, and Jev optimally responds even to its own *wrong* written-back predictions (24/26).
• In ALFWorld, ordered-form tasks (steps spelled out) → 100% sub-goal accuracy; goal-form ("put a *clean* knife in drawer", washing never mentioned as a verb) → 33%, errors all jumping to "put". Writing its own answer back: 68%→86%; code adding the correct sub-goal: 92%; rewriting the task as ordered form: 6%→73%. Failure is not missing knowledge but a boundary inside a single call's structure.

The prescription: code owns the simulation, Jev owns the evaluation

Four ways code can supply the simulation: 1. Direct statement ("Column will play X") → 97%+. 2. Write-back of the model's own prediction (two calls) → 94%. 3. Lookahead: run every candidate command in a copy of the game and append its outcome and newly unlocked commands. Misleading ALFWorld steps 36%→79%, lure rate 56%→21%; full unseen episodes solved: 31% → 87%. The same lookahead given to a naive word-overlap rule solves only 18 episodes — the value comes from Jev's evaluation, not the information alone. 4. Physics simulation (the skill library above).

Cost: ~94M input tokens ≈ $4 total at then-current API pricing; simulation on a single server's CPUs.

Honest limitations

• Cross-option relations (e.g., dominance) remain invisible; code must handle them.
• It obeys the order actions appear in task text even when "after" contradicts it (24% vs 69%).
• Only one model/API tested; whether the boundary holds for all judgment models (or shifts with reasoning enabled) is unknown.
• Lookahead used an exact environment copy and depth 1; deeper search costs grow quickly. Without a simulator, write-back works only when Jev can make the prediction itself (goal-form's 33% requires a learned world model or a larger reasoner).
• The boundary is architectural, not a matter of scale or data.

Positioning

The paper maps onto model-free vs model-based control (Daw et al. 2005): Jev alone is the model-free half; code-supplied simulation completes the model-based half. It inverts Tree-of-Thoughts/RAP/LATS-style designs where the same model proposes and scores futures, giving SayCan-style "code lists actions, model scores" pipelines a principled trust condition. Together with Mingbird (arXiv:2610.02001, the *potential layer*: capabilities locked in the harness) and mechanistic-consistency work (arXiv:2610.01842, the *hallucination layer*), this paper forms the *boundary layer* of a complete capability map: know where brute force stops working and architectural division of labor should take over.

Reference

• Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang. *Code Owns the Simulation, Jev Owns the Evaluation.* arXiv:2610.01834, 2026.