Soup: Fine-tuning 8B Models on a 4GB GPU with Layer Streaming — and an Honest Release Gate

Forum topic · ✨步子哥 · 2026-08-15

Summary

Soup is an open-source fine-tuning tool that fine-tunes 8B-parameter LLMs on a 4GB laptop GPU (RTX 3050) using a technique called layer streaming: frozen base model weights stay in CPU RAM or NVMe and are streamed layer-by-layer through VRAM during forward passes, so only LoRA adapters and one decoder layer occupy GPU memory at any time. On Llama-3.1-8B with NF4 quantization, it reports 119.6 tok/s training throughput at a 3.32GB peak, bit-exact versus standard resident execution, independently reproduced on an H100. Equally notable is Soup's release gate (soup ship): v0.73.2 openly documented three evaluation bugs — a parser misjudging correct answers, an extractor failing on \boxed{} formats (mini_mmlu score corrected from 0.423 to 0.731), and a missing over-refusal axis — and introduced a --noise-floor option to account for GPU greedy decoding non-determinism (score spread 0.015–0.020 across runs). The project exemplifies honest engineering: acknowledging potentially stale benchmarks and measuring evaluation noise instead of chasing fake significance.

Overview

Soup (https://github.com/MakazhanAlpamys/Soup) uses a "layer streaming" technique to fit fine-tuning of 8B models onto a 4GB laptop GPU — but the more compelling story is its release gate: v0.73.2 proactively acknowledged three evaluation bugs and quantified the non-determinism of GPU greedy decoding.

A Counterintuitive Number

On an RTX 3050 laptop GPU with 4GB VRAM, conventional wisdom says fine-tuning beyond 3B models causes OOM errors.

Soup's answer: 8B model, 119.6 tok/s, 3.32GB peak.

This is not post-quantization inference — it is fine-tuning: QLoRA training with batch size 4 and sequence length 512.

The trick is a technique called layer streaming.

Layer Streaming: Kicking Frozen Weights Out of VRAM

When fine-tuning an 8B model, only the LoRA adapter's few million parameters need gradients. But the entire 8B model normally has to sit in VRAM because the forward pass reads it.

Layer streaming keeps the frozen base model on CPU RAM or NVMe, feeding it to the GPU layer by layer during the forward pass.

Rather than loading the whole model into VRAM, it works like a pipeline: decoder layer 1 finishes and is evicted, then layer 2 enters VRAM, and so on. Only one layer occupies VRAM at any moment.

It's like reading a book — you don't photocopy the entire book into your brain; you turn pages one at a time.

The Cost

Layer streaming isn't free. The cost is speed — each layer must be read from RAM/SSD to VRAM, making PCIe bandwidth the bottleneck. But Soup's measurements show: on an RTX 3050 4GB, Llama-3.1-8B + NF4 quantization + LoRA reaches 119.6 tok/s with a 3.32GB peak.

Crucially, results are bit-exact compared to resident execution. Not an approximation, not trading accuracy for memory — bit-level identical. Independently reproduced on an H100: 113.00 tok/s, same 3.32GB peak.

An Honest Footnote

> The tok/s figure was measured on v0.72.2, before the v0.73.0 correctness repair that cost −4.8% at 32B; it has not been re-run on a 4 GB card since.

Translation: the speed number was measured on v0.72.2; v0.73.0 fixed a correctness bug at a cost of 4.8% slowdown at 32B, and the 4GB-card number hasn't been re-measured.

This kind of honesty — proactively declaring that a number may be stale — is rare in open source. Most projects put flattering numbers at the top of the README and bury corrections in the changelog.

The Real Highlight: The Release Gate

Soup's v0.73.2 release notes carry the title:

> the release gate stops lying in both directions

Soup has a soup ship command that answers one question: did this model actually get better, or did I break it? It runs seven evaluation suites and decides whether a release is allowed based on results.

v0.73.2 found the gate was lying in three ways:

Bug 1: mini_ was punishing correct answers

The mini_ suite scored a model with 40/40 correct answers at 0.225. Why? The model omitted one closing brace, the parser fell back to an inner object, and the scorer declared "missing outer key" — a false failure.

The model was right, the parser broke it, and the gate blamed the model.

Bug 2: mini_mmlu scored Llama-3.1-8B at 0.423

Lower than a 0.5B model. The extractor didn't recognize \boxed{C}-style output, and the prompt never asked the model to output a letter.

After the fix: 0.423 → 0.731.

The model knew the answers, but the evaluator couldn't read its output format.

Bug 3: No benign-prompt axis

The original gate had no "benign prompt refusal rate" dimension. A model that refused all requests (including benign ones) scored byte-identically to a normal model across all seven suites — the gate couldn't distinguish them.

Fix: a new mini_over_refusal suite, paired with the safety suite so neither can be gamed alone.

GPU Greedy Decoding Is Not Deterministic

v0.73.2 also introduced a --noise-floor N option, for a hard-nosed reason:

> Greedy decoding is not deterministic on GPU — same model, no adapter, five runs spread 0.015–0.020 against a 0.05 threshold, and four of six paired deltas in that session sat inside the floor.

Same model, same data, same code, five runs on GPU — scores vary by 0.015–0.020. Meanwhile your "significant improvement" threshold is 0.05.

This means if you see a model go from 0.80 to 0.82 and credit the LoRA adapter, it may just be GPU noise.

--noise-floor N first runs the base model N times to measure the noise distribution, then any delta smaller than that spread doesn't count as significant.

This is the right approach. Most LLM evaluation reports never consider it — they run the base model once, the tuned model once, see a 2-point difference, and publish a paper.

Two Threads, One Project

Soup hits two conceptual threads at once:

"Division of labor beats unification": layer streaming separates what needs gradients (adapters, kept in VRAM) from what doesn't (frozen base, evicted from VRAM).

"Evaluation blind-spot law": all three bugs were evaluation blind spots — the evaluator couldn't parse output formats (bugs 1+2), an evaluation dimension was missing (bug 3), noise was unmeasured (noise floor). Soup's honesty lies in writing these bugs into its release notes rather than pretending they don't exist.

Why This Matters

Soup isn't just "fine-tune an 8B model on a 4GB card" — it's a complete fine-tuning engineering practice:

One-line YAML config, no SSH needed
Automatic GPU detection, batch size selection, and quantization
A release gate that tells you whether the model really improved
Proactive disclosure of potentially stale benchmark numbers
Measured GPU noise and rejection of fake significance

Most fine-tuning frameworks only care about "can you train" — Soup cares about "how do you know you didn't break anything after training."

The second question matters more, because the failure mode of LLM fine-tuning isn't OOM — it's a model that looks better while getting worse on dimensions your evaluation never measures.

Soup's v0.73.2 patched three such dimensions. The next release will patch more. That's what engineering maturity looks like — not "we have no bugs," but "we proactively found where our evaluation was lying."

---

Project Info

GitHub: https://github.com/MakazhanAlpamys/Soup
Install: pip install "soup-cli[train]"
Stars today: 303 (trending on 2026-08-15)
Highlights: 8B model fine-tuning on 4GB GPU + a release gate that doesn't lie