LoRA Distillation of DeepSeek-V4-Pro's Chain-of-Thought into Qwen3.6-35B-A3B: A Qualitative Leap for Agent Orchestrators

Key points

• Problem. Qwen3.6-35B-A3B (MoE, 35B total / ~3B active) is cost-efficient but weak as an Agent orchestrator: overthinking, execution-reasoning mismatch, ~12 empty answers per loop, and ~60s end-to-end latency. At ~20K context it can hit ~100% loop-output bug rate.
• Teacher. DeepSeek-V4-Pro (1.6T total / 49B active, 1M context, MCPAtlas 73.6%) plus DeepSeek-V4-Flash for fast, multi-style responses. The teacher's thinking-on mode emphasizes accurate switching between reasoning and action, not longer thinking, fitting the ReAct paradigm.
• Method. LoRA fine-tuning on Qwen3.6-35B-A3B (MoE, 128/256 experts, top-k 8) with r=64, α=128, lora_target=all, 1,842 orchestration-specific traces, BF16 precision, DeepSpeed ZeRO-2 on 2× A100 80GB, ~1 day 15 hours, final loss ~0.85. Distillation targets the *thinking mode* (how to decompose, dispatch, verify), not raw knowledge.
• Results.
• End-to-end orchestration latency: 60s → 26.6s (2.3× faster)
• Empty answers per loop: 12 → 1 (-92%)
• False verification: non-zero → 0
• GPQA-Diamond: +7.6pp
• Lynn 4-gate eval: g1 style 4.83/5, 100% cliché-free, g2 tool-call regression +0.00pp, g2 academic holdout +8.33pp, g3 baseline +40.00pp
• Variants. Pro (deep reasoning, long research/coding/math) and Flash (short/medium chains, tool calls, coding agents); both ~65GB BF16 / ~20GB Q4_K_M.
• Deployment notes. Required inference params: thinking=on, temperature=0.6 (never greedy), top_p=0.95, top_k=20, no presence_penalty > 1.0. Available paths:
• ModelScope: modelscope download --model Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill
• Ollama/llama.cpp with Q4_K_M GGUF (~20GB, single 24GB GPU), launch with --jinja, -c 32768
• vLLM BF16 (recommended), 2× tensor parallel, --max-model-len 8192
• SGLang dev-cu13 for NVFP4 with --quantization compressed-tensors, --reasoning-parser qwen3, --tool-call-parser qwen3_coder
• Comparison with R1-Distill-Qwen-32B. R1 distills general math/reasoning on ~800k samples via full SFT; this project targets Agent orchestration on 1,842 samples via lightweight LoRA, producing halved orchestration time and zero false verification rather than generic benchmark gains.
• Lessons. Agent systems should split roles: orchestrators need fast decision-making (this distillate), workers need deep reasoning (V4-Pro/Flash), tools stay deterministic. Good Agents iterate between reasoning and action quickly, not deeply.
• Limitations. No public benchmark scores (AIME/GSM8K/HumanEval+); SGLang requires qwen3_coder parser for <tool_call> blocks; Chinese-dominant training data yields mixed CN/EN long outputs; Q4_K_M vs NVFP4 differences stem from chat-template design, not quantization quality; do not load quantized checkpoints via raw AutoModelForImageTextToText (causes silent random init / garbled output); vLLM v0.19.0 may not work on sm_121, prefer SGLang dev-cu13; keep ≥128K context for thinking capacity, though orchestration tasks are usually <8K.

References

• Model: https://modelscope.cn/models/Merkyor/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill
• Lynn project: https://github.com/MerkyorLynn/Lynn
• Base model: https://github.com/QwenLM/Qwen3
• Teacher: https://github.com/deepseek-ai/DeepSeek-V4
• ReAct paper: Yao et al., 2022, "ReAct: Synergizing Reasoning and Acting in Language Models"