EnigmaForge: When Benchmarks Stop Handing You the Question

> A forum editor's structured summary of the Chinese post "No One Asks Questions in the Archive Room" — an in-depth walkthrough of the EnigmaForge benchmark (arXiv:2609.30144, Daniel Eisner, 24 Sep 2026).

Key points

Why a new benchmark

• Traditional benchmarks suffer from saturation: frontier models cluster at 95–99 on MMLU, GSM8K, HumanEval, destroying discriminative power.
• Contamination: since training corpora swallow the public web, test items leak into training data, making high scores ambiguous. The result is the endless "benchmark treadmill."
• EnigmaForge's answer is not a harder exam but leaving the exam room entirely.

Design philosophy: a puzzle generator, not a puzzle bank

• 120 puzzle families (abstract logical skeletons) × 5 literary realizations (Victorian correspondence, ship logs, butler records, commercial receipts) = 600 instances, tagged with 6 difficulty levels.
• Generated, not collected: fresh generation from seeds structurally resists contamination — "you cannot memorize a flowing river."
• Proven, not trusted: each puzzle is verified by a DPLL (SAT) engine to have a unique solution, and ships with an ablation certificate — every clue is brute-force checked to be load-bearing; removing any clue yields a second solution.
• Three strictly matched conditions: explicit (question stated), formal (formalized constraints only), and implicit (story only, no question given). The gaps between conditions are the real measurement.

Anatomy of a puzzle

• Each puzzle has a surface goal (e.g., reconstruct the shift schedule) and a true goal (a final action decision), with rules that can be superseded mid-story — models must track rule versions.
• Knowledge bridges: some constraints require external world knowledge (e.g., that Shakespeare wrote Hamlet).
• Distractors: clues that are true and support real hypotheses but are useless for the solution.
• The stakes: pure forward chaining reaches 0.998 accuracy, while naively copying stated facts already yields 0.737 F1 — the benchmark is deliberately built to hunt down "repeater" models.

Core findings (25 models, 600 instances, 17,400 ratings)

• Intuition scores range from 4 to 100 — a 22x gap. Most models, handed only the documents, never realize there is a puzzle at all.
• Fact recovery varies only 1.6x: reading comprehension has converged; the divergence is in *discovery*. Notably, the model ranked 2nd in fact-recovery F1 ranked only 14th on Intuition.
• Behavioral outliers: one model is completely insensitive to whether a question is given; another actually performs better without an explicit question — the stated problem acts as a cognitive fixation that narrows attention.

The safety-filter problem

• Multiple models were blocked by their own content filters before reaching the puzzle — period-style letters and logs looked "suspicious." Refusals score zero.
• The paper's pointed conclusion: *any benchmark that counts refusals as failure is quietly measuring the filter's behavior, not the model's capability.* Rejection rates are condition-dependent, injecting directional noise into evaluations.

The bottleneck has moved: from perception to decision

• Two new metrics: Discovery retention (58–102; performance without a stated question vs. with one) and Earned decisions (28–88; correct actions among instances where the model correctly recovered the facts).
• Since fact recovery has converged but correct decisions still span a ~3x range, the frontier bottleneck has shifted from seeing to deciding — implying new training paradigms (RL, world models) rather than more scale.

The lone exception: GPT-6 Sol

• The only model to essentially close the explicit/implicit gap: discovery retention 102 (it performs slightly *better* with no question given) and earned decisions 88, leading all world-reconstruction metrics.
• Whether this lead is a replicable recipe or architectural luck is left open — but one counterexample is enough to prove the 22x gap is not a law of nature.

The closing thought

EnigmaForge ultimately measures something closer to the essence of intelligence: whether a system can grow its own goals in a world that never hands it an exam. The real world piles up ambiguous, dusty, contradictory information with no annotated questions. Most frontier models remain far — 22x far — from finding the question on their own. But the distance is now being measured, and once measured, it is only a matter of time before it is closed.

Reference

• Eisner, Daniel. "EnigmaForge: The Question Is Hidden in the Story." arXiv:2609.30144, submitted 24 Sep 2026.