Chain-of-Thought Distillation: Teaching Small Language Models to Reason Like Giants

This post summarizes and analyzes a November 2025 paper by Cong-Thanh Do, Rama Doddipatla (Toshiba Europe Cambridge Research Laboratory) and Kate Knill (University of Cambridge): *"Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models"* (arXiv:2511.05184v1).

Background: Emergent Reasoning and the Cost Barrier

Reasoning—the use of logic, evidence, and knowledge to draw conclusions—appears to be an emergent capability in large language models: it reliably appears only when model scale reaches tens of billions of parameters (Wei et al., 2022; Fu et al., 2023). But large models like GPT-4, Claude, and Gemini demand heavy compute, making them unsuitable for latency-sensitive, edge, or cost-constrained scenarios.

Small models (Qwen-1.8B, Llama2-7B, TinyLlama-1.1B) offer 3–5x faster inference, lower latency, edge deployability, and 80–90% lower per-inference cost—but struggle with complex reasoning. The paper's question: can chain-of-thought (CoT) reasoning be distilled into these small models?

Method: White-box KD + CoT

Traditional "black-box" distillation only exposes the teacher's final text output. This work uses white-box knowledge distillation, where the student sees the teacher's full output probability distribution, combined with CoT rationales:

1. Data: 1.84M training samples from the CoT-Collection dataset (Kim et al., 2023), covering 1,060 tasks (multiple-choice QA, extractive QA, formal logic, NLI, arithmetic), with rationales generated by OpenAI Codex. 2. Forward pass: Samples go through both teacher (Qwen-7B or Llama2-13B-Chat) and student (Qwen-1.8B, Llama2-7B, or TinyLlama-1.1B). 3. Distillation loss: Minimize Kullback-Leibler divergence between teacher and student output distributions:

\(D_{KL}(P||Q) = \sum_i P(i) \log\frac{P(i)}{Q(i)}\)

4. Backpropagation: Gradient updates the student at every reasoning step.

The crucial difference from prior white-box KD: rationales are kept in the training data rather than filtered out.

Training setup

• Built on the MiniLLM framework (Gu et al., 2024); 20,000 steps, 10 epochs, LR 5e-6, batch size 16, τ=1, max prompt length 512
• LoRA (r=32, alpha=32, dropout=0.1) when Llama2-7B is the student
• ~30 hours on an NVIDIA A100 80GB GPU
• Evaluation: 3-shot CoT prompts, temperature 0.2, on BIG-Bench-Hard (27 challenging tasks)

Results on BBH

| Model | Baseline | +KD | +KD+CoT | |---|---|---|---| | Qwen-1.8B | 17.77% | 23.10% (+30.00%) | 24.44% (+37.54%) | | Llama2-7B | 39.44% | 39.22% (−0.56%) | 41.50% (+5.22%) | | TinyLlama-1.1B | 27.96% | 26.48% (−5.29%) | 29.23% (+4.54%) |

Notable per-task gains for Qwen-1.8B+KD+CoT:

• Formal fallacies: 11.6% → 36.4% (+213.79%)
• Hypernym adjective ordering: 9.2% → 51.2% (+456.52%)
• Tracking shuffled objects (5 obj.): 2.8% → 15.2% (+442.86%)
• Temporal sequences: 14.8% → 28.4% (+91.89%)

But Boolean expressions regressed (43.2% → 34.8%, −19.44%). Dyck-language results also improved strongly (TinyLlama: 12.8% → 27.6%, +115.62%).

Key Insights

1. CoT is a catalyst for KD: On both Llama2 students, plain white-box KD *failed* to beat the baseline, yet KD+CoT delivered solid gains. Explicitly modeling reasoning steps matters more than mimicking final answers. 2. Task specificity: Gains are large on structured, multi-step reasoning tasks; tasks demanding precise symbolic manipulation (Boolean expressions) can be hurt by verbose natural-language rationales. 3. Architecture sensitivity: Qwen and Llama2 responded differently to CoT signals (e.g., on Disambiguation QA they moved in opposite directions), suggesting distillation strategies should be architecture-aware. 4. Noise transfer risk: In a sports-understanding example, KD+CoT led TinyLlama to hallucinate that "touchdown" applies to baseball—teachers with unsharp probability distributions can transmit faulty associations. 5. Qualitative wins: KD+CoT students produced measurably better reasoning chains, e.g., correctly identifying the free time slot in temporal sequences and avoiding a hallucinated translation in significant-error detection (Kassel vs. "Tossel").

Inference Speed

Distilled students retain identical parameter counts, so their speed is unchanged—while drastically outpacing their teachers:

• TinyLlama-1.1B: ~20% of Llama2-13B-Chat inference time
• Qwen-1.8B: ~28% of Qwen-7B
• Llama2-7B: ~53% of the teacher

Practical Takeaways

• For edge/latency-constrained deployment, KD+CoT is a strong recipe for boosting small-model reasoning.
• BBH effectively differentiates reasoning-quality differences among distilled models.
• High-quality CoT datasets (like CoT-Collection) can be more cost-effective than scale alone.
• Evaluate teacher-student architectural compatibility and filter teacher rationales to limit noise.

Future Directions

The authors' framework suggests promising extensions: adaptive CoT length, multi-teacher distillation, cross-modal CoT, explainability for high-stakes domains, and continual learning of new reasoning patterns after deployment.

Reference

Do, C.-T., Doddipatla, R., & Knill, K. (2025). *Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models*. arXiv:2511.05184v1.