Jev and the "Decision Model" Narrative: Four Gates of Fact-Checking

> A 2026.10.04 deep-research report from the zhichai.net forum. The original long-form article asked the right questions but built its conclusions on a broken evidence chain. This report re-verifies every statistic with independent scripts and does not rely on secondhand accounts.

One-line verdict. The article's core conceptual move — separating "does the decision-model category have value?" from "is Jev actually as intelligent as claimed?" — is its most valuable contribution. But the evidence chain breaks in three places: n=100 samples cannot support 4–7 pp gaps; the baseline was weakened by a generated-text-parsing protocol; and the "bearish for giants" thesis was overturned by OpenAI itself within 14 days of Jev's release.

Jev's real position. A price and speed crusher whose capability stably trails the strongest frontier models by 3–7 percentage points — but not a single row of the article's 12-row benchmark table passes statistical testing.

Key points

• Gate 1 — Numbers are real, effects are not. Recomputed two-proportion z-tests (n=100, two-sided): Jev's four "wins" (FOLIO +4pp p=0.471; CommonsenseQA +5pp p=0.249; Winogrande +7pp p=0.0299; ProofWriter +12pp p=0.0090) all fail Bonferroni correction (α′ = 0.00417). The only clearly significant difference is a 21-point loss on MuSR-team-assign (52% vs 73%, p=0.0022).
• The design is paired, the test was not. Both models ran the same 100 questions (seed=42, fixed manifest), so McNemar's test is required; the article reports no discordant-pair counts (b, c) — a structural flaw, not a sample-size problem.
• Sampling pools differ by 50×. FOLIO's 100 questions cover 49% of its source set; ProofWriter's cover 0.8%. Treating these as equal evidence is misleading. Minimum detectable differences at n=100 range from ~8.6pp (95% accuracy) to ~19.8pp (50% accuracy); detecting 4pp would require ~1,251 items per group.
• Cost claim inflated both ways. "2 cents vs ¥5" is 35× after conversion, not 250×. Real figures: Jev's actual per-call cost (5k tokens at $0.042/1M) is $0.00021 — the article overstates it ~95×, and $0.02/call is physically impossible under a 64k context cap; the Qwen figure is overstated ~1,300×. Third-party estimates of the cost gap: 17–250×.
• Latency: half true. The 0.3 s Jev figure matches four independent measurements (236–276 ms medians). The 13 s figure for Qwen is not physically plausible for a 3.3B-active-parameter MoE in non-thinking mode; realistic speedups are 6–18× server-side / 1.7–3.9× end-to-end. AIMLAPI: "Jev wasn't 100x faster than any model."
• The central misreading of official data. Jev's 67.8% is an equal-weight average of four workflows in which it ranks 3rd, 7th, 8th, and 4th — first place zero times — yet the article read "parity on average" as "parity with the frontier." Independent checks (AIMLAPI; OpenRouter n=3,080 paired bootstrap, CI [2.3, 4.4]pp) confirm frontier models lead by ~7 points, p<0.001.
• Official headline numbers mix models. TypeSafe's "193.6× faster" uses Sonnet 5 timing while "444.6× cheaper" uses Opus 5 cost; the vendor concedes both are "on the higher end" and that its workflow reference labels were LLM-derived, not human-annotated.
• Memorization: unresolved by the paper itself. arXiv 2609.37647 shows option-shuffling probes are not Jev-specific, but explicitly states they "cannot distinguish memorization of question-answer pairs from a genuine ability," calls training exposure to MMLU "the most plausible explanation," and defers the decisive numeric-perturbation test to future work. Jev has no technical report, weights, or data disclosure.

Gate 2 — The baseline is rigged

• The jeveval harness has Qwen generate {"label":..., "probabilities":{...}} as text and parse it (adapters/openai_compatible.py:38-39,56-75), while Jev reads native API probabilities — apples vs oranges. Calibration (ECE/Brier) comparisons built on it are invalid, and calibration is precisely Jev's headline feature.
• Disabling chain-of-thought is honest and enforced in code, but Qwen3's own ablations show the largest drops (−22pp on AIME25, −22.3pp on LiveCodeBench) exactly in the region of the chosen benchmarks; conclusions hold only for "same-price direct-answer tier," a caveat the article omits.
• Tool hygiene: 3 commits, 1 contributor, no license, no committed real Jev-vs-Qwen results, null reference prices, and an unpinned jev-latest model alias — which the vendor's own docs warn silently invalidates tuned thresholds. Credit is due for seed+SHA-256 manifests, caching discipline, and honest data-quality notes.

Gate 3 — Category value is real, but savings leak downstream

• "Decision models" are not new: monoT5 rerankers (2019), FrugalGPT / RouteLLM / Hybrid LLM (2023–24), Chow's rule (1970), SelectiveNet (2017), Rewarding Doubt (2025). Forbes columnist Anthony Maio: "RLCD is a made-up phrase by this startup firm."
• The "System 1" framing uses labels Evans & Stanovich (2013) explicitly discontinued; speed is a derived correlate, not the defining feature (which is autonomy and working-memory independence).
• Most damaging counterexample: on 272 real tickets, Jev's calibrated error was 0.05 for ticket typing ($0.03 vs $5.63, the ~170× saving holds) but 0.22 for product area (claims of ≥90% confidence were only 79% accurate). At a 90% threshold Jev covered just 61% of traffic vs Claude's 85% at 92.8% accuracy — human review claws back much of the savings.

Gate 4 — "Bearish for the giants" misses the target

The thesis was retrodiction, not prediction: OpenAI entered the same space within 14 days of Jev's launch.

Errata (selected from 15 items)

1. Cost ratio is 35×, not 250× (missed currency conversion). 2. Per-call cost figures inflated ~95× (Jev) and ~1,300× (Qwen). 3. Latency ratio is 6–18×/1.7–3.9×, not 43×. 4. "±5pp margin of error" is wrong — SE varies from ±9.8pp CI half-width at p=50% to 0 at p=100%. 5. Official data read as "ties frontier" when Jev never ranks first on any single workflow. 6. Probes presented as "ruling out memorization" when the paper says the opposite.

Bottom line: a correct problem decomposition, a category-level thesis worth keeping, and a capability claim that collapses under its own spreadsheet — corrected by the author's own recomputation rather than received wisdom.

*Source: zhichai.net forum deep-research post, 2026-10-04. Referenced: arXiv 2609.37647, arXiv 2609.29429, arXiv 2503.02623, arXiv 2307.12950, evals.typesafe.ai.*