Paper Overview
- Research Area: NLP
- Authors: Cheng Qian, Wenting Zhao, Liangwei Yang
- Release Date: 2026-08-13
- arXiv: 2508.03418
- Setup: Four representative Theory-of-Mind (ToM) benchmarks. Each builder model uses 5% of the data as a validation set and refines its harness over multiple iterative rounds; the final harness is evaluated on the full test set.
- Headline Result: Test-time capability transfer is highly effective — average target-model performance rises from 0.49 to 0.91, nearly doubling.
- Where the gains come from:
- Offloading unstable model reasoning into deterministic code.
- Benchmark-specific routing.
- Strict answer-format enforcement.
- Notably, gains are *not* driven by encouraging broader target reasoning or wider sampling.
- Further findings:
- The builder model's reasoning effort monotonically increases harness quality.
- Platform effects are modest relative to the builder's own capability.
- Weaker target models benefit the most.
- arXiv: https://arxiv.org/abs/2508.03418
Summary
Recent distillation research typically transfers large-model capabilities to smaller models by updating the smaller model's parameters through teacher forcing, online distillation, and related training-time methods. This paper asks whether such transfer can instead occur at test time.
The authors study strong-to-weak scaffolding: a stronger *builder* model designs inference-time harnesses (tooling such as prompts, code, and routing logic) that help a weaker *target* model solve tasks more reliably, without any parameter updates on the target.
Key Points
Implications
Inference-time harness design is positioned as an important complement to traditional training-time distillation: strong models can transfer cognitive structures to weak models without retraining, by externalizing reasoning into well-designed scaffolding.