Overview
Soup (https://github.com/MakazhanAlpamys/Soup) fine-tunes 8B models on a 4GB laptop GPU via "layer streaming," and its v0.73.2 release candidly admits three evaluation bugs — and measures GPU greedy-decoding nondeterminism.
The counterintuitive number
- Hardware: RTX 3050 laptop GPU, 4GB VRAM
- Result: 8B model fine-tuning, 119.6 tok/s, 3.32GB peak (QLoRA, batch size 4, seq length 512)
- This is fine-tuning, not just quantized inference
- Layer streaming separates "what needs gradients" (adapter, in VRAM) from "what needs storage" (frozen base, outside VRAM)
- The three bugs were all evaluation blind spots: output-format blindness, missing dimensions, unmeasured noise
- Soup is a complete fine-tuning workflow: one-line YAML config, no SSH, auto GPU detection, auto batch size, auto quantization
- The failure mode of LLM fine-tuning isn't OOM — it's a model that looks better while degrading on dimensions your evaluation never measures
- GitHub: https://github.com/MakazhanAlpamys/Soup
- Install:
pip install "soup-cli[train]" - 303 stars (featured 2026-08-15)
How layer streaming works
Only the LoRA adapter (a few million parameters) needs gradients during fine-tuning, yet the full 8B base weights normally must reside in VRAM for the forward pass. Layer streaming instead keeps the frozen base model in CPU RAM or NVMe and feeds decoder layers to the GPU one at a time — like reading a book page by page rather than copying the whole book into memory. At any moment only one layer occupies VRAM.
The cost: speed. PCIe bandwidth becomes the bottleneck. But measurements show Llama-3.1-8B + NF4 + LoRA at 119.6 tok/s on the 4GB card, and results are bit-exact with fully-resident runs — not an approximation. An independent H100 run reproduced 113.00 tok/s, same 3.32GB peak.
An honest footnote from the README:
> The tok/s figure was measured on v0.72.2, before the v0.73.0 correctness repair that cost −4.8% at 32B; it has not been re-run on a 4 GB card since.
This proactive disclosure of possibly-stale numbers is rare in open source.
The release gate stops lying in both directions
soup ship runs seven evaluation suites and decides whether a model can be released. v0.73.2 found the gate was lying in three ways:
1. mini_ parser bug: a model scoring 40/40 correct was given 0.225 because a missing closing brace made the parser fall back to an inner object. The model was right; the parser was wrong.
2. mini_mmlu scored Llama-3.1-8B at 0.423 — below a 0.5B model — because the extractor didn't recognize \boxed{C} format and the prompt never asked for a letter. After the fix: 0.423 → 0.731.
3. Missing benign-prompt axis: a model refusing all requests was byte-identical to a normal model on the seven suites. Fixed by adding a mini_over_refusal suite paired with safety suites so neither can be gamed alone.
GPU greedy decoding is not deterministic
> Greedy decoding is not deterministic on GPU — same model, no adapter, five runs spread 0.015–0.020 against a 0.05 threshold.
The new --noise-floor N option runs the base model N times, measures the noise distribution, and treats any delta smaller than that spread as not significant. Most LLM evaluation reports never account for this.