English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Between 53 Milliseconds and 30 Seconds: ThinkJev Decides When an LLM Should Think

Forum topic · 小凯 · 2026-11-11

Summary

Many production LLM tasks—ticket routing, policy checks, field extraction—reduce to picking one letter from a fixed set of options, where what matters is the probability of each option rather than prose. A new paper, ThinkJev, from three researchers at Fujian Boss Software (arXiv:2610.37732), connects two answering modes of a hybrid-thinking model (Qwen3.5 at 4B/9B/27B) without training any new weights: a 53 ms direct read, and a 30.3 s median chain-of-thought. Both modes share the same prompt KV cache, and option-letter probabilities can be read at any thinking checkpoint, so one thought serves every budget. Admission to the fast path is gated by a statistical procedure using Clopper-Pearson one-sided bounds rather than raw confidence. At a 5% target error rate, the 27B model routes 24.5% of 8,359 test decisions directly with a measured 3.9% error, reaching 81.6% accuracy versus 82.7% for always-think while saving 72% of thinking tokens. On the independent JevBench, it triggered thinking on only 29.4% of questions while matching two commercial reasoning APIs on accuracy (flagged exploratory). The paper honestly reports limitations: single sampling per decision, calibration valid only on its development distribution, and checkpoint forward passes excluded from cost.

Between 53 Milliseconds and 30 Seconds

Which queue should a ticket go to, whether a message crosses a policy red line, whether an extracted field passes validation—many LLM tasks ultimately reduce to one choice: pick a letter from a few fixed options and hand it to code. These problems share a trait: the answer isn't an essay, it's the probability of each option.

Given one such question, the same model can answer two ways. Direct read: one forward pass, 53 milliseconds. Think-then-read: let the model write a reasoning trace first, median 30.3 seconds and 1,038 thinking tokens. The latter is far more accurate—especially on arithmetic, counting, and date comparison, where single-step answers go wrong. The problem is those 30 seconds: paying them even on easy questions is too expensive.

A paper released in late October, ThinkJev, hooks both answering modes onto the same model. The authors are three researchers at Fujian Boss Software; the code is open source. It trains no new weights—all the effort goes into deciding *whether* to think and *when to stop*.

One model, two systems

The base is a hybrid-thinking model (the paper uses Qwen3.5 at 4B, 9B, and 27B), which natively supports two output modes: answer directly, or think first. ThinkJev treats the former as System 1 and the latter as System 2. The key design: both share the same prompt's KV cache. Easy questions read the answer directly; when thinking is needed, generation continues from the cache—the cost of escalation is only the thinking itself, with zero prefill recomputation.

Even better: "readable at any time." Option answers are letters (A, B, C...), and letter probabilities can be read out at any step of the thinking process. So the same trace, read at 100 tokens, is one "thinking budget"; read at 500 tokens, another. The paper proves in an appendix that checkpoints are equivalent to thinking budgets: one thought equals thinking through every budget at once.

There's a trap in intermediate reads, handled carefully: the model becomes "overconfident" as it thinks. Raw confidence rises faster than accuracy—the 27B model's calibration temperature climbs from 1.61 (direct) to 4.07 (after full thinking). The longer it thinks, the more its bare confidence brags. So each checkpoint gets its own temperature calibration before intermediate reads are trustworthy (calibration error squeezed to 0.02–0.04).

Who answers directly? Statistics decides

The intuitive approach for routing is trusting the model's own confidence: if direct confidence is high, let it through. ThinkJev instead uses a statistical procedure: on a labeled development set, candidate thresholds are tested from strictest to loosest, each verified with a Clopper-Pearson one-sided upper bound that "routed decisions' error rate does not exceed target," stopping at the first failure. The passing threshold is carried as-is to the test set—how much gets routed and how many errors follow are guaranteed.

At a 5% target error rate, the 27B model routed 24.5% of 8,359 test decisions with a measured 3.9% error. More interesting are the other two: 4B dared to route only 6.5%, and 9B routed zero. The statistical gate doesn't care how confident you are, only whether the evidence holds. 9B's direct error rate couldn't support any threshold, so the gate released nothing. That's the honesty of this procedure: it would rather let System 2 handle everything than make a promise it can't keep.

Routed decisions answer in milliseconds. The rest go to System 2, reading while thinking: whichever checkpoint's confidence crosses the line, stop there. Result: 81.6% accuracy versus 82.7% for "always think to the end," saving 72% of thinking tokens. Paying 1.1 percentage points for a 70% cut in thinking cost.

Why intuition isn't enough

The paper's most informative section takes the intuitive approach—confidence-based routing—and shoots it down. Confidence-based escalation saves little. The reason is in the data: a substantial share of System 1's least-confident decisions are ones thinking *can't* fix—multi-step reasoning that's wrong stays wrong no matter how long it thinks. Escalating them costs money and gains no accuracy.

The paper also tries three routers: a zero-shot router (the model judges "can this be answered at a glance"), a probe router reading hidden states, and an n-gram router. The first two rank surprisingly as well as the statistical gate, but all strategies remain more than half short of ideal routing—the best normalized area on 27B is only 0.491. No single signal currently carries "which questions deserve thinking" alone. Combining the gate's confidence with zero-shot scores (a post-hoc logistic regression) beats everything across all models.

A matchup against commercial APIs

The paper's own test set can't compare against commercial services (no API calls), so it borrows an independent benchmark: JevBench, 231 public typed decision questions, with Jev 1.13.0 and two commercial thinking APIs (DeepSeek V4.1 Flash, GPT-5.6 Luna) as baselines. ThinkJev's 27B triggered thinking on only 29.4% of questions while matching both commercial APIs on accuracy—the paper honestly labels this an exploratory conclusion. Two caveats: JevBench's total score factors in speed and cost, but only accuracy is reported here; escalated decisions take seconds, while Jev itself answers sub-second. Thinking less didn't lose accuracy, but didn't win speed either.

Limitations are laid out candidly: one thinking sample per decision, no second-seed fallback; temperature and thresholds are valid only on the paper's own development distribution—new domains need recalibration, and changed budgets require re-tuning temperature; cost counts only thinking tokens, excluding the small forward passes at intermediate checkpoints.

Look back at those numbers: a 53 ms direct read, a 30-second think, and in between, a gate guarded by Clopper-Pearson bounds. On JevBench, the 27B thought on only 29.4% of questions while matching two commercial thinking APIs on accuracy. The model didn't get smarter—"which questions don't need thinking" got a statistical endorsement for the first time.

---

*Source: ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions (arXiv:2610.37732, 2026-10-27, Xiaolu Ding, Kai Yang, Dongdong Liu, Fujian Boss Software; code at github.com/ding-xiaolu/ThinkJev). All figures from the paper: 53 ms forward pass and 30.3 s / 1,038-token median thinking (Section 5.8, single H200); 8,359 test decisions in three blocks (A daily 2,626 / B public reasoning 2,487 / C multi-step 3,246); 24.5% routed at 3.9% measured error under a 5% target (4B 6.5% / 9B 0.0%); 81.6% vs 82.7% and 72% thinking-token savings; temperature 1.61→4.07 and ECE 0.017–0.036; router comparison 0.491 (Section 5.7); JevBench 231 questions and 29.4% thinking share (Section 5.9, exploratory labeling from the paper); limitations from Section 6.*

Tags

#llm#efficient-inference#chain-of-thought#adaptive-computation#statistical-guarantees#confidence-calibration#kv-cache#thinkjev

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637040