Homebrewed SLMs: Training Your Own Language Model from Scratch on a Single Consumer GPU

"Building a Data Center" Is No Longer a Prerequisite

In 2026, what does it take to train a language model? If your answer is "thousands of H100s and millions of dollars," Pedro Brom's paper will make you think again.

Brom, a mathematician from the Federal Institute of Brasília, trained an 11-12M parameter language model from scratch on a single consumer GPU — including tokenizer training, model training, evaluation, and even a preregistered comparative experiment protocol. The entire process takes minutes, not days, let alone months.

The paper is titled Homebrewed SLMs. The word "homebrew" is well chosen: it evokes the 1970s Homebrew Computer Club — the era when Jobs and Wozniak assembled the Apple I in a garage. Brom is saying: the garage era of language models has arrived.

This Paper Is Not a Methods Paper

The paper states upfront: it proposes no new methods. It is a tutorial, a reproducible experimental report, a Python package, and two Jupyter notebooks.

But precisely this "nothing new" quality is what makes it valuable. Current language model research has an implicit assumption: training models is "big tech's job," and academia can only fine-tune and analyze. Brom demonstrates through practice that training a useful model from scratch has a far lower barrier than you'd imagine.

The key is not model size but being "good enough." An 11-12M parameter model won't write poetry, but it can handle domain-specific tasks — like the Brazilian federal law domain tested in this paper.

From Scratch Means Really From Scratch

The tutorial's complete pipeline includes:

1. Corpus: Brazilian Portuguese legal texts 2. Tokenizer: trained from scratch, no reliance on existing tokenizers 3. Architecture: a unified network supporting two generation modes 4. Training: single consumer GPU, minute-scale training 5. Evaluation: not just loss, but end-to-end domain task evaluation 6. Comparison protocol: preregistered hypothesis testing, not "run first, look at results later"

Points 5 and 6 are the paper's most underrated contributions. Most "I trained a small model" work stops at reporting loss curves. Brom designed four legal-domain tasks:

• T1 Fill-in-the-blank: completing blanks in legal provisions
• T2 Insertion/Substitution: judging whether a provision has been tampered with
• T3 Tamper detection: identifying modified text
• T4 Continuation: continuing legal provisions

These tasks require only text — no legal expertise — yet genuinely test whether the model "understands" the structure of legal language.

Two Generation Modes: Autoregressive vs Masked Diffusion

A technical highlight: the same network supports two generation modes.

• Autoregressive mode: left-to-right token-by-token generation, the standard GPT paradigm
• Masked diffusion mode: starting from fully masked, progressively revealing tokens

Both modes use the same network, differing only in inference. This lets you compare generation strategies on the same trained model.

Experimental findings:

• Autoregressive mode peaked at 64 epochs, then began memorizing training data (overfitting)
• Masked diffusion mode was still improving at 256 epochs
• On the test set, diffusion mode had likelihood better than autoregressive by 0.06 bits per byte — within the preregistered minimum effect boundary

0.06 bits/byte sounds small, but on language model likelihood metrics it's a meaningful difference. More importantly, the two modes' optimal training durations differ completely — you cannot use the same hyperparameters for both.

Honest Defect Reporting

The most impressive part of this paper is not the results but the honesty. Brom devotes substantial space to "defects only revealed by measurement":

The Gradient Precision Sentinel Bug

The paper designed a "precision sentinel": comparing the same gradient under fp16 and fp32 to ensure fp16 training introduced no numerical error. Calibration passed (error < 0.01), but in the formal experiments, all 12 runs exceeded the preregistered 0.05 threshold — with a maximum of 7.11.

The cause? A bug in the sentinel itself. Both backward passes ran in training mode, each independently sampling dropout masks — measuring not "precision differences" but "the distance between two random masks." Setting both to eval mode brought errors back to normal range.

The lesson: verification tools themselves need verification. If you design a precision checker, the checker's own bugs can invalidate all check results.

Losing to 6-gram on Continuation

On the "continuing published law" task, the 11M parameter model lost to a simple 6-gram language model. Brom doesn't hide this result but directly discusses where the ceiling lies for small models in the trade-off between memorization and generalization.

Beating a 630M Parameter Model on Tamper Detection

But on the "detect tampered text" task, the 11M model beat 160M and 630M public models — without ever having seen the "decrees" genre during training. This shows small models can "punch above their weight" on specific tasks; the key is matching tasks to training data.

From "Scoring" to "Scoring + Calibration"

The paper also performs calibration analysis — whether the model's confidence matches its actual accuracy. Most small model work doesn't report calibration, but Brom found:

• Calibration was good on the "fill-in-the-blank" task
• But the model was overconfident on "continuation" — it thought it could continue text when actual continuation quality was poor

This aligns with the "lost to 6-gram" finding: the model knows it has "seen" similar text, but doesn't know it can't "retain" enough context to continue it.

Why This Paper Matters

Homebrewed SLMs' value is not in how good its model is — an 11M parameter model is of course far behind GPT-4 in absolute capability. Its value lies in three things:

1. Proof of Accessibility

It demonstrates through practice that training a useful language model from scratch doesn't require a data center. A consumer GPU, a domain-specific corpus, and a rigorous evaluation protocol are enough. This matters enormously for academia and small teams — you don't need to rent an H100 cluster to do language model research.

2. Evaluation Methodology

Preregistered comparison protocols are extremely rare in ML papers. Most papers are "run, then tell a story about the results"; Brom registers hypotheses and minimum effects first, then runs experiments. This practice is standard in clinical trials but almost nobody does it in ML. The paper also reports negative results — diffusion's 0.06 bits/byte advantage was right at the preregistered minimum effect boundary, not a "significant advantage" but "just met the preset standard."

3. Defect Transparency

The honest report of the precision sentinel bug is far more valuable than a paper claiming "everything works." It tells readers: even if you design verification mechanisms, the mechanisms themselves can fail. This awareness — "verification needs to be verified" — is severely lacking in current ML experimental culture.

Echoing the "Evaluation Blind Spot Law"

This paper connects to a cross-paper theme: the Evaluation Blind Spot Law — what you measure is what models optimize; what you don't measure is your blind spot.

• τ-bench (discussed in the Concordat paper): only checks final database state, not process compliance
• Perplexity (discussed in the distillation paper): can't see post-distillation loop collapse
• Homebrewed SLMs: if you only look at loss curves, the model seems to keep improving; but the fact that "continuation loses to 6-gram" is only revealed by end-to-end tasks

Brom's evaluation protocol — multiple tasks + calibration + preregistration — is a direct response to evaluation blind spots: don't look at just one metric, don't look only at training curves, don't test only tasks you're good at.

Limitations

The paper's limitations are clear:

1. Only tested Portuguese legal domain: generality needs validation in more domains 2. 11-12M parameters is very small: whether conclusions generalize to 100M+ small models is unknown 3. No new methods: as a tutorial paper, it doesn't advance the methods frontier — but its value was never in methodological innovation

But if you're a researcher or small team wanting to train a language model from scratch, this paper is currently the best starting point. Code and notebooks are publicly available.

---

Paper link: https://arxiv.org/abs/2610.40533

HTML version: https://arxiv.org/html/2610.40533v1

Open-source code: https://github.com/pcbrom/homebrewed-slms

Reference implementation: based on Karpathy's nanoGPT