Saluki 27B Deconstructed: A 7.9GB '27B' Model, Bolted-On Vision, and the 262K Context in the Headline

Forum topic · 小凯 · 2026-10-09

Summary

A viral Chinese deployment tutorial promotes Saluki 27B, a 2-bit quantized Qwen3.8-27B derivative claimed to offer 7x compression, multimodal capability, and 262K context on a 16GB GPU. Analysis of the full pipeline shows: Saluki is ConwayResearch's IQ2-mix quantization of ISTA-DASLab's GSQ-RCO GGUF of Qwen3.8-27B (54GB fp16 down to 7.89GB, a 6.84x ratio). Multimodal ability is inherited from the base model via a separate mmproj file the tutorial omits, and 262K is only the base config's max_position_embeddings — running that context needs ~16GB of KV cache on top of the model. The genuinely notable result: Saluki scores 88 vs 84 on its frozen Underdog Bench tool-calling tasks, beating the full-precision model, while math benchmarks drop ~15 points (AIME 2025: 79.2 vs 96.7), averaging 96% retained across nine benchmarks.

Saluki 27B Deconstructed: A 7.9GB "27B", Bolted-On Vision, and the 262K in the Headline

On October 8, a deployment tutorial went viral in Chinese tech-tutorial circles. It promoted a model called Saluki 27B, claiming it compresses Qwen3.8-27B "7x" down to 7.9GB, includes "multimodal" ability and "262K context," runs agent tool calling on a 16GB GPU, and ships with direct cloud-drive links. I verified all three selling points. Conclusion up front: the compression is real, the multimodal claim needs quotation marks, the 262K is a paper spec inherited from the base model — and the most interesting thing about this model is not mentioned in the tutorial's title at all.

The Lineage

Saluki is not trained from scratch. It is the end of a three-layer pipeline. At the top is Qwen's official Qwen3.8-27B: 54GB of fp16 weights with three architectural features — a SigLIP-style vision tower, a multi-token prediction (MTP) speculative decoding head, and hybrid attention (only 16 of 64 layers use traditional full attention; the other 48 are linear attention). The second layer is ISTA-DASLab's quantized release — the group of Dan Alistarh, where GPTQ originated — using their GSQ-RCO method to produce a non-uniform quantized GGUF of Qwen3.8-27B, allocating precision per tensor by sensitivity. That upstream has 1.49M downloads on Hugging Face. The third layer is Saluki itself: ConwayResearch (a company whose on-device model line is all dog-named — Woof, Bark, Husky, Saluki) uploaded an extreme IQ2-mix, 2-bit quantization on the early morning of October 8: 7.89GB, Apache 2.0, with a clean chain.

So the "7x compression" is just quantization: 54 / 7.89 = 6.84, roughly 7. There's no magic — 2-bit quantization gives exactly this compression ratio. The real technical content lies in precision allocation and post-quantization tuning, but the tutorial title framed it as a property of the model itself, leading readers to think it's a new architecture.

Multimodal, In Quotes

Saluki's model card is explicit: the main file is text-only; an additional 0.6–0.9GB vision file is needed for images. The vision capability is inherited from the base Qwen3.8, shipped as an external mmproj file that must be downloaded separately and mounted at launch. The tutorial never mentions this file — an environment built strictly from it cannot do image understanding. It's the same playbook as the earlier FrogNano case: base-model-inherited capability appears in the artifact, and the tutorial title markets it as a model feature.

262K: The Weakest Claim of the Three

262,144 is a real number, but it lives in the base Qwen3.8-27B config as max_position_embeddings — a paper value that quantized weights neither change nor are responsible for. What matters is whether it runs. Thanks to the hybrid architecture, only 16 full-attention layers need a traditional KV cache: 262K context costs roughly 16GiB of KV (fp16), plus the 7.9GB model — far beyond a 16GB GPU. The tutorial's own launch command uses -c 32768 — a 32K context with just 2GiB of KV, which is the real world of one 4080 SUPER. Between paper specs and deployment reality stands the size of a graphics card.

What Saluki Is Actually Good At

Tool calling survived quantization — and even improved. On the model card's self-built Underdog Bench (120 tasks frozen from the Berkeley Function Calling Leaderboard), Saluki scores 88 vs 84 for full-size Qwen3.8-27B; parallel tool calling is 42 vs 35. A 2-bit quantized version beating the original by 4 points on core agent capability is a rare "negative quantization tax" in this space. The cost is on the same table: competition math retains only 82–85% of the original (AIME 2025: 79.2 vs 96.7), SWE-bench 30 vs 33, MuSR 67.5 vs 79.6 — averaging 96% across nine benchmarks.

The model card's honesty deserves note: task sets were frozen before testing any model, the different harnesses for self-built vs public scores are disclosed, and even the small-sample caveat is stated ("120 tasks is a modest test; a few points may be run-to-run variance"). Two ambiguities remain: how 88 vs 84 was achieved — quantization strategy, calibration data, or post-quant repair — is unexplained; and the opponent it claims to beat "by 18 points on its own benchmark," Bonsai 2, is the true incumbent of this space. prism-ml's Ternary-Bonsai-2-27B has 4.39M cumulative downloads, 287x Saluki's two-day total of 15K. A challenger winning on its own exam, with freezing disclosed, deserves serious consideration — but until the harness is open-sourced, reproducibility is limited.

Two Footnotes

Renaming economics: Conway released Underdog-27B-1.0 on September 30 (315 downloads, ignored), iterated 1.1 on October 4 (still quiet). After rebranding as a dog-named Saluki positioned as "Qwen3.8 compressed 7x": 15K downloads and 144 likes in two days. Same model, new name, new fate.

Ecosystem speed: the community converted Qwen3.8's MTP head into a speculative decoding GGUF (Kujira's MTP-GGUF, thousands of downloads) the same day; Conway launched its own underdog-llama-server runtime distribution within a week — moving from model to full stack.

As for the Quark/Thunder cloud-drive links: that's not Saluki's achievement, but mirror infrastructure born of domestic access conditions to Hugging Face. The tutorial's 183 reads versus 15K HF downloads in two days show most users already use HF; the tutorial is just a fallback for those who don't.

Two numbers to leave with: 88 vs 84, and 7.89 vs 54. The first is a 2-bit quantized model outscoring the full-size original on tool calling; the second is the 6.84x compression ledger. Whether to accept a 15-point math drop for an improved score depends on whether your agent needs to do math — and on this question, the model card is far more honest than the tutorial title.