Can an 86M-parameter GPT-2 do arithmetic? In an era of hundred-billion-parameter models like GPT-4 and Claude, the question sounds like asking whether a bicycle can outrun a bullet train. But two researchers from Institut Teknologi Bandung and IT Del offer a striking answer: yes — if you teach it correctly. Their secret is not a bigger model but an Indonesian math pedagogy called GASING.
GASING: Left-to-Right, a Natural Fit for Language Models
GASING is a arithmetic teaching method widely used in Indonesian basic education. Its core idea is simple: compute from left to right. Traditional column arithmetic proceeds right-to-left (units → tens → hundreds), requiring low digits to be resolved before high ones. GASING reverses this: for 37 + 25, first compute 30 + 20 = 50, then 7 + 5 = 12, finally 50 + 12 = 62.
For humans, left-to-right matches how we read and speak numbers. For language models, it is a perfect match: autoregressive models generate tokens left to right, so GASING's computation order aligns exactly with token generation — no "looking back" required.
Three Learning Stages: From Rote Memorization to Mental Math
The researchers serialized each GASING operation step into natural-language chain-of-thought (CoT) data and trained an 86M GPT-2 decoder from scratch — no reinforcement learning, no reward model, just plain next-token prediction. The model showed three clear stages:
- Stage 1 — Rote memorization. The model mechanically copies CoT steps without understanding the logic, like a schoolchild who can recite the multiplication table but not apply it.
- Stage 2 — Procedural computation. The model internalizes the procedural path and executes steps correctly. Attention-ablation experiments show it depends on intermediate CoT results — computing each step honestly.
- Stage 3 — Mental calculation. The model suddenly "gets it" and produces intermediate results without step-by-step computation. Residual-stream probing and logit lens analysis reveal an "associative mental calculation" ability — like humans, it memorizes through repetition.
86M Beats Much Larger Models
The final result: the 86M model achieved over 80% accuracy on unseen arithmetic problems, comparable to language models with far more parameters. The key is "teaching it right" — GASING decomposes operations into steps aligned with the model's generation order, avoiding operations that violate its architectural instincts.
The Tokenizer Matters Too
The researchers used the TOBA tokenizer, a syllable-based tokenizer designed for agglutinative Indonesian. Compared to BPE, TOBA splits digits more finely, making each digit token semantically purer and further easing arithmetic learning.
Broader Implications
Beyond arithmetic, the work raises a fundamental question: should we teach AI the way we teach humans? The mainstream approach is brute force — more data, bigger models, stronger rewards. GASING's success shows that if you understand the model's "cognitive structure" (autoregressive, left-to-right) and design teaching methods aligned with it, small models can do what big models do.
Paper link: https://arxiv.org/abs/2606.05106