Paper Overview
Field: Machine Learning Authors: Jundong Hu, Shekar Ramachandran arXiv: 2610.00025
Summary
Agent harnesses increasingly want to run small language models (SLMs) on the microtasks around a frontier large language model (LLM) planner: auto-approving shell commands, writing memory, selecting tools, ranking past turns. The paper asks whether off-the-shelf SLMs meet practitioner-defined thresholds and, when they fail, why — and whether quantization changes the answer.
The authors build a benchmark of 4 such microtasks with fixed prompts and automatic metrics, each with a pre-specified threshold \(\tau\) anchored to a cheap non-LLM baseline and a CI-aware eligibility rule (a configuration passes only if its confidence bound clears \(\tau\)).
Key Findings
- Eligibility gap: Sweeping Qwen3 0.6/1.7/4/8B at their best (FP16, greedy, one frozen prompt, no tuning), 0 of 16 (4 tasks × 4 models) configurations pass, verified by inspecting raw outputs and parser behavior.
- Failure diagnosis: Logprob-based decision-threshold diagnostics separate capability deficits from failures resolvable by decoding thresholds (4 status categories).
- Quantization: 4-bit quantization (RTN/GPTQ/AWQ) has size-dependent impact but qualifies no configuration; the gap tracks model scale more than precision.
- Reproducibility & robustness: Results reproduce on Llama-3.x (12/12 ineligible) and are robust to anchor selection and prompt wording, including 3 neutral paraphrases (0/112 eligible).
- Practical implication: Place SLMs behind baselines whose CI-reachable thresholds they can clear, and use them only where baselines fall short — e.g., a 4B reranker outperforms BM25 shortlisting (+0.047 [0.020, 0.073]) while still not being certified eligible itself.
Original Abstract (excerpt)
> Agent harnesses increasingly want to run small language models (SLMs) on the microtasks around a frontier large language model (LLM) planner: auto-approving shell commands, writing memory, selecting tools, ranking past turns. We ask whether off-the-shelf SLMs meet practitioner-defined thresholds and, when they fail, why, and whether quantization changes the answer. We build a benchmark of 4 such microtasks with fixed prompts and automatic metrics, each with a pre-specified threshold \(\tau\) anchored to a cheap non-LLM baseline and a CI-aware eligibility rule (a configuration passes only if its confidence bound clears \(\tau\)). Sweeping Qwen3 0.6/1.7/4/8B at their best (FP16, greedy, one frozen prompt, no tuning), we find an eligibility gap: 0 of 16 (4 tasks × 4 models) configurations ...
*Auto-collected on 2026-10-05.*