Qwythos-9B: Distilling Claude-Style Reasoning into a 9B Model with 1M Context on 4GB VRAM

Forum topic · 小凯 · 2026-06-25

Summary

Qwythos-9B is an open-source reasoning model built on the Qwen3.5-9B (abliterated/uncensored variant) architecture, post-trained on over 500 million high-quality reasoning traces from Claude Mythos/Fable. The core idea is that a model's reasoning style—self-questioning, step-by-step decomposition, backtracking, and metacognitive method selection—can be transferred independently of raw knowledge storage, so a 9B model can learn how to think rather than what to know. Reported benchmark gains include +34 on MMLU and +30 on GSM8K Strict versus the base model. The model uses YaRN (RoPE-based position interpolation with attention temperature scaling) to extend context to a theoretical 1.04M tokens, supports native Function Calling and Vision multimodality, and runs locally: a Q4_K_M quantized version fits in 4GB VRAM at 16K context. The post also discusses limitations: 9B-scale knowledge boundaries, degraded attention precision beyond trained context lengths, quantization loss, missing safety alignment, and unresolved legal questions about distilling from Claude API outputs. Overall, Qwythos signals the democratization of reasoning-capable AI through local deployment.

Qwythos-9B is an open-source reasoning model built on the Qwen3.5-9B architecture, post-trained via distillation on 500M+ high-quality reasoning traces from Claude Mythos/Fable. It extends context to 1.04M tokens with YaRN, supports native Function Calling and Vision, and a Q4_K_M quantized version runs on just 4GB of VRAM.

Model link: https://ollama.com/richardyoung/qwythos-9b-abliterated

Key point 1: Why reasoning style can be distilled

Traditional view: capability = parameters x data. Distillation experiments (DeepSeek-R1 onward) suggest knowledge (facts, language patterns) and reasoning patterns (how to decompose, verify, backtrack) are relatively separable.
Qwythos trains on complete chain-of-thought trajectories—not Q&A pairs. Key features: self-questioning ("let me check again..."), step-by-step decomposition, backtracking on errors, and metacognition (choosing a method before executing).
Unlike standard fine-tuning (teaching correct answers), reasoning distillation teaches the thought process leading to answers.

Key point 2: The uncensored base

The base is an "abliterated" (safety-alignment-removed) Qwen3.5-9B. The author argues heavy alignment can suppress open-ended chain-of-thought, so removing constraints helps learn exploratory reasoning styles.

Key point 3: Likely training pipeline

1. SFT on Claude-style reasoning trajectories (input: question; target: full reasoning + answer). 2. DPO/RLHF ranking multiple reasoning paths to reinforce self-verification and backtracking (speculation—the blog doesn't detail the method). 3. YaRN context extension: interpolates RoPE positions into the trained range, applies attention temperature scaling, and uses NTK-aware per-frequency handling—expanding 32K→1M without retraining.

Key point 4: Benchmark interpretation

| Benchmark | Gain | Reading | |---|---|---| | MMLU | +34 | Knowledge density improved via trajectories | | GSM8K Strict | +30 | Rigorous multi-step reasoning learned | | GSM8K Flex | +19 | Gains in creative problem-solving too |

Large GSM8K gains suggest reasoning-pattern transfer works—plain Q&A distillation would likely move MMLU more than GSM8K.

Key point 5: 1M context in theory vs practice

YaRN maps unseen positions into the trained range; but attention dilution, retrieval precision decay, and KV-Cache size (tens of GB at 1M tokens) mean the practical sweet spot is 32K–128K. The blog itself concedes 1M is a theoretical ceiling.

Key point 6: Function Calling + Vision for local deployment

Native tool calling (Qwen3.5 standard) enables offline agents with local data privacy.
Vision support allows local screenshot/OCR/chart analysis without cloud upload.

Key point 7: Honest limitations

| Aspect | Reality | |---|---| | Scale | 9B cannot match 70B/400B on complex reasoning | | Knowledge | Distills reasoning patterns, not full knowledge | | Safety | Uncensored—may emit harmful content | | Long context | Attention precision degrades past trained lengths | | Quantization | Q4_K_M may hurt complex reasoning | | Data provenance | Source of 500M Claude traces undisclosed; possible ToS issues |

The post raises whether this is legitimate distillation or a terms-of-service violation if Claude API outputs were used for training a competing model.

Deployment reference

| VRAM | Quant | Context | |---|---|---| | 4GB | Q4_K_M | 16K | | 6GB | Q5_K_M | 32K | | 8GB | Q6_K | 64K | | 12GB | Q8_K_M | 128K | | 16GB | BF16 | 256K | | 24GB | MTP-BF16 | 512K |

Supported frameworks: llama.cpp, llama-server, OpenWebUI, Cherry Studio, OpenClaw.

Takeaway

Qwythos-9B is less a breakthrough than a signal: reasoning style is transferable, 1M-class context is no longer exclusive to frontier models, and a "thinking" model now runs on a laptop GPU—accelerating the democratization of AI capability.