How a 4B Model Caught Up with a 27B Teacher: Half Task Practice, Half Copying Homework

A 4B small model scored 51.8% on 394 data-analysis questions. In the same exam room, its 27B teacher scored 64.0% — a 12.2-point gap.

YouTuber PromtEngineer recently ran an experiment: without swapping the brain, how much of that gap can training alone recover? The answer: more than half. But it takes two steps, and the second step matters more than the first.

The testbed: two NVIDIA DGX Spark units, LoRA fine-tuning of Qwen3.5-4B, with GRPO as the algorithm. The question bank is SmolDataEnvs — 5,394 Kaggle-style data-analysis tasks. Each task provides real data files; the model works in a bash terminal and writes its answer to a file. Grading is done by an independently written strict checker. Code, checker, and training scripts are all open-sourced on GitHub (repo created two days ago, Apache-2.0).

Act One: Random Practice Was Almost Useless

How GRPO learns is worth explaining in one sentence: the model attempts the same task several times, and the difference between the successful and failed attempts within the group is the learning signal. The corollary is direct: if a group is all-correct or all-wrong, there is no within-group difference, no signal — that round was wasted.

After 40 steps of randomly sampled tasks, the score was 54.1% with a p-value of 0.29 — statistically indistinguishable from no training. The cause was found: with random tasks, 55% to 70% of groups were all-correct or all-wrong. Whether the model aces a task or is completely lost, no gradient is produced.

Act Two: Task Selection

Since the signal only exists in tasks the model sometimes gets right, filter first. Out of 999 candidate tasks, each was attempted 4 times, keeping only the 240 tasks the model sometimes solved. Training on this batch of borderline tasks for 120 steps took 19.1 hours.

The score rose to 58.4%, with a paired McNemar test p=0.001 — a genuine gain. That recovered 6.6 of the 12.2-point gap: 54%.

What It Actually Learned

This is the most counterintuitive part of the experiment. RL did not add data-analysis ability.

Two statistics: episodes ending without an answer written dropped from 99 to 72; wrong-format answers (asked for a name, gave a number) fell from 17 to 6. Translation: the 4B model could already solve many tasks, but often turned in a blank sheet or wrote the answer in the wrong format. What RL mainly fixed was "finishing the job and formatting the answer," not "knowing more."

This also explains the ceiling. Before training, the 4B got 62.3% of training tasks right in a single attempt, and 72.9% within four attempts. RL can only grind "solves it within four tries" into "solves it first try" — it cannot create ability that never existed. That's why four different subsequent training routes all plateaued around 58%. The ceiling is the base model itself.

Act Three: Copying Homework

After plateauing, tactics changed. The 27B teacher solved these tasks with full solution traces; those traces were used to SFT-distill the 4B, and then RL was applied on top.

On the test set, the combined model scored 58.6% — nearly tied with pure RL's 58.4%. The real difference appeared on unseen tasks: on 87 brand-new dataset tasks, pure RL scored 58.6% while distillation-plus-RL reached 62.1% (p=0.04 vs the untrained starting point). Against the teacher's 64.4%, that recovers 80% of the gap.

Distillation imports solutions the 27B knows and the 4B doesn't; RL then polishes those solutions. On the test set the two look identical, but on new tasks the difference shows — what was learned is the method, not the specific questions.

The Grader and Padding Answers

The benchmark's built-in grader is lax: it grabs the first number in the answer — if the correct answer is 42, answering "42, yes" still passes. Using that as the training reward, the model would quickly learn to stuff numbers into answers.

The author's solution: write a separate strict grader as the reward, keep the lax one in the loop with zero weight, purely as an alarm. Post-run checks confirmed no answers were padding; everything the lax grader caught was a genuine near-miss.

The reward is the grader. A lax grader teaches the student to pad — a lesson that holds repeatedly in robot training, in RLHF, and in this experiment.

Hardware Scars

Three hard-won lessons, essential for reproduction: log probabilities must match between the training side and the vLLM sampling side — without flash-linear-attention installed, long responses get almost no training weight; changing repetition_penalty from 1.05 to 1.0 reduced per-token probability divergence from 0.06 to 0.012; DGX Spark's GB10 unified memory doesn't cleanly report OOM, so you must monitor available memory yourself. All versions are pinned in versions.txt — TRL, vLLM, and model snapshots, no exceptions.

Wrap-Up

This experiment breaks "small model catching up with big model" into two countable parts: of the 12.2-point gap, 6.6 points came from training on the right tasks — mostly converting blank sheets into submitted ones; the rest required the teacher to show its work.

Some background: RL environments are becoming the new scarce resource — TechCrunch covered Silicon Valley's collective bet on environments last September. This question bank comes from the FineEnvs project, and Xiaomi's MiMo-V2.6 has also opened up its RL environments. Compute and models keep getting cheaper, but gradeable tasks are always in short supply.

GitHub: github.com/PromtEngineer/smoldataenvs-grpo-spark (15 stars, two days) Question bank: huggingface.co/datasets/FineEnvs/SmolDataEnvs (5,394 tasks, 7,434 downloads) Model cards: Qwen3.5-4B / Qwen3.5-27B, both Apache-2.0