Between 53 Milliseconds and 30 Seconds
Which queue should a ticket go to, whether a message crosses a policy red line, whether an extracted field passes validation—many LLM tasks ultimately reduce to one choice: pick a letter from a few fixed options and hand it to code. These problems share a trait: the answer isn't an essay, it's the probability of each option.
Given one such question, the same model can answer two ways. Direct read: one forward pass, 53 milliseconds. Think-then-read: let the model write a reasoning trace first, median 30.3 seconds and 1,038 thinking tokens. The latter is far more accurate—especially on arithmetic, counting, and date comparison, where single-step answers go wrong. The problem is those 30 seconds: paying them even on easy questions is too expensive.
A paper released in late October, ThinkJev, hooks both answering modes onto the same model. The authors are three researchers at Fujian Boss Software; the code is open source. It trains no new weights—all the effort goes into deciding *whether* to think and *when to stop*.
One model, two systems
The base is a hybrid-thinking model (the paper uses Qwen3.5 at 4B, 9B, and 27B), which natively supports two output modes: answer directly, or think first. ThinkJev treats the former as System 1 and the latter as System 2. The key design: both share the same prompt's KV cache. Easy questions read the answer directly; when thinking is needed, generation continues from the cache—the cost of escalation is only the thinking itself, with zero prefill recomputation.
Even better: "readable at any time." Option answers are letters (A, B, C...), and letter probabilities can be read out at any step of the thinking process. So the same trace, read at 100 tokens, is one "thinking budget"; read at 500 tokens, another. The paper proves in an appendix that checkpoints are equivalent to thinking budgets: one thought equals thinking through every budget at once.
There's a trap in intermediate reads, handled carefully: the model becomes "overconfident" as it thinks. Raw confidence rises faster than accuracy—the 27B model's calibration temperature climbs from 1.61 (direct) to 4.07 (after full thinking). The longer it thinks, the more its bare confidence brags. So each checkpoint gets its own temperature calibration before intermediate reads are trustworthy (calibration error squeezed to 0.02–0.04).
Who answers directly? Statistics decides
The intuitive approach for routing is trusting the model's own confidence: if direct confidence is high, let it through. ThinkJev instead uses a statistical procedure: on a labeled development set, candidate thresholds are tested from strictest to loosest, each verified with a Clopper-Pearson one-sided upper bound that "routed decisions' error rate does not exceed target," stopping at the first failure. The passing threshold is carried as-is to the test set—how much gets routed and how many errors follow are guaranteed.
At a 5% target error rate, the 27B model routed 24.5% of 8,359 test decisions with a measured 3.9% error. More interesting are the other two: 4B dared to route only 6.5%, and 9B routed zero. The statistical gate doesn't care how confident you are, only whether the evidence holds. 9B's direct error rate couldn't support any threshold, so the gate released nothing. That's the honesty of this procedure: it would rather let System 2 handle everything than make a promise it can't keep.
Routed decisions answer in milliseconds. The rest go to System 2, reading while thinking: whichever checkpoint's confidence crosses the line, stop there. Result: 81.6% accuracy versus 82.7% for "always think to the end," saving 72% of thinking tokens. Paying 1.1 percentage points for a 70% cut in thinking cost.
Why intuition isn't enough
The paper's most informative section takes the intuitive approach—confidence-based routing—and shoots it down. Confidence-based escalation saves little. The reason is in the data: a substantial share of System 1's least-confident decisions are ones thinking *can't* fix—multi-step reasoning that's wrong stays wrong no matter how long it thinks. Escalating them costs money and gains no accuracy.
The paper also tries three routers: a zero-shot router (the model judges "can this be answered at a glance"), a probe router reading hidden states, and an n-gram router. The first two rank surprisingly as well as the statistical gate, but all strategies remain more than half short of ideal routing—the best normalized area on 27B is only 0.491. No single signal currently carries "which questions deserve thinking" alone. Combining the gate's confidence with zero-shot scores (a post-hoc logistic regression) beats everything across all models.
A matchup against commercial APIs
The paper's own test set can't compare against commercial services (no API calls), so it borrows an independent benchmark: JevBench, 231 public typed decision questions, with Jev 1.13.0 and two commercial thinking APIs (DeepSeek V4.1 Flash, GPT-5.6 Luna) as baselines. ThinkJev's 27B triggered thinking on only 29.4% of questions while matching both commercial APIs on accuracy—the paper honestly labels this an exploratory conclusion. Two caveats: JevBench's total score factors in speed and cost, but only accuracy is reported here; escalated decisions take seconds, while Jev itself answers sub-second. Thinking less didn't lose accuracy, but didn't win speed either.
Limitations are laid out candidly: one thinking sample per decision, no second-seed fallback; temperature and thresholds are valid only on the paper's own development distribution—new domains need recalibration, and changed budgets require re-tuning temperature; cost counts only thinking tokens, excluding the small forward passes at intermediate checkpoints.
Look back at those numbers: a 53 ms direct read, a 30-second think, and in between, a gate guarded by Clopper-Pearson bounds. On JevBench, the 27B thought on only 29.4% of questions while matching two commercial thinking APIs on accuracy. The model didn't get smarter—"which questions don't need thinking" got a statistical endorsement for the first time.
---
*Source: ThinkJev: One Thought, Every Budget for Calibrated Typed Decisions (arXiv:2610.37732, 2026-10-27, Xiaolu Ding, Kai Yang, Dongdong Liu, Fujian Boss Software; code at github.com/ding-xiaolu/ThinkJev). All figures from the paper: 53 ms forward pass and 30.3 s / 1,038-token median thinking (Section 5.8, single H200); 8,359 test decisions in three blocks (A daily 2,626 / B public reasoning 2,487 / C multi-step 3,246); 24.5% routed at 3.9% measured error under a 5% target (4B 6.5% / 9B 0.0%); 81.6% vs 82.7% and 72% thinking-token savings; temperature 1.61→4.07 and ECE 0.017–0.036; router comparison 0.491 (Section 5.7); JevBench 231 questions and 29.4% thinking share (Section 5.9, exploratory labeling from the paper); limitations from Section 6.*