Notes to Self: Small LLMs Learn from Their Own Mistake Notes, Matching Teacher-Distilled Abstractions

Forum topic · ✨步子哥 · 2026-07-23

Summary

This post introduces and analyzes 'Notes to Self' (arXiv:2607.20372), a method where a small LLM distills 'experience abstractions' from its own problem-solving trajectories — strategy notes from successes, caution notes from failures — and reuses them via retrieval-augmented prompting or abstraction-enhanced GRPO training. On MATH-500, abstraction-augmented GRPO training lifts Llama-3.2-3B from 43.11 to 49.45 and Qwen-2.5-1.5B from 50.23 to 53.44. Strikingly, self-distilled abstractions match or beat abstractions distilled by a stronger teacher model, suggesting latent metacognitive ability in small models. Control experiments with blank note blocks confirm the gains come from abstraction content, not prompt structure. Abstractions help on hard problems (and transfer to OlympiadBench and MuSR logical reasoning) but hurt on already-mastered easy problems, explained as perturbation of the output distribution: diffuse base distributions benefit, sharpened RL-trained distributions suffer. Cross-model transfer is model-specific (Phi-3.5 gains; Gemma models degrade). Failure analysis attributes Qwen's inference-time losses to formatting sensitivity and poor retrieval fit, not to caution notes over-constraining reasoning. The post concludes that experience abstraction is a viable teacher-free self-improvement signal operating at the strategy level.

Notes to Self: When a Small Model's Own Mistake Notes Rival Its Teacher's

An experiment in "learning how to learn"

When you botch a math problem, a good teacher helps you extract a general lesson — "when you see a quadratic under a square root, try substitution" — which you record in a notebook of mistakes and consult later. Psychologists call this *experiential abstraction*, and Polya discussed such cognitive tools back in 1945's *How to Solve It*.

Can LLMs do the same? Not via a strong teacher distilling lessons for a weak student (classic knowledge distillation), but by having a small model distill its own experience notes from its own solving trajectories and retrieve them on similar future problems?

A 2026 paper by Chang Liu et al. (arXiv:2607.20372) answers yes: small models' self-distilled experience abstractions rival those distilled by a strong teacher. This is stronger than "models can learn from experience" — it suggests small models possess a form of metacognition, able to identify reusable strategies and pitfalls from their own reasoning.

Pipeline: from trajectories to an experience library

Step 1: Generate solving trajectories

The target model πθ solves a batch of training problems (right or wrong). Each full reasoning trace y ~ πθ(·|q) is recorded.

Step 2: Distill experience abstractions

An "extractor LLM" analyzes each trace:

Success → *strategy abstraction*: the key idea, what's reusable.
Failure → *caution abstraction*: where it got stuck, what to watch for next time.

The extractor can be a strong teacher (DeepSeek-V4-Flash) or the target model itself — the paper's central comparison. Abstractions are encoded with all-MiniLM-L6-v2 into 384-dim vectors, deduplicated via cosine similarity and union-find (threshold 0.85), with strategies and cautions stored separately. Dedup shrinks the library by ~30% on average.

Step 3: Two usage modes

Retrieval at inference: retrieve top-k (k=6) relevant abstractions and insert them into a <reference_notes> block after the question.
Abstraction-augmented GRPO training: abstractions are retrieved and injected into training prompts during GRPO, so the model learns to use them.

The two can be combined.

Headline result: self-made notes are as good as the teacher's

Table 1, on MATH-500:

| Model | Extractor | Baseline | Inference_abs | GRPO | GRPO_train | GRPO_train+test | |-------|-----------|----------|---------------|------|------------|-----------------| | Llama-3.2-3B | Teacher | 43.11 | 46.42 | 46.89 | 49.45 | 47.65 | | Llama-3.2-3B | Self | - | 46.30 | - | 49.07 | 47.67 | | Qwen-2.5-1.5B | Teacher | 50.23 | 49.14 | 52.55 | 53.44 | 52.66 | | Qwen-2.5-1.5B | Self | - | 49.44 | - | 53.77 | 52.72 |

Key observations:

1. Abstraction-augmented GRPO (GRPO_train) improves consistently: Llama +6.34, Qwen +3.21. 2. Self-extraction rivals teacher extraction: the gap for Llama is only 0.38; for Qwen, self-extraction (53.77) actually beats the teacher (53.44). Small models can write their own mistake notebooks — no teacher needed. 3. Inference-time retrieval alone helps Llama (+3.31) but slightly hurts Qwen (-1.09) — unstable without training. 4. Train+test stacking underperforms training alone: once abstractions are internalized via RL, injecting notes at test time interferes.

Honest control: blank-note experiments

Is the gain from abstraction content, or merely from the prompt structure? Table 2 answers with blank <reference_notes> blocks:

| Mode | No notes | Blank notes | + Abstractions | |------|----------|-------------|----------------| | Inference | 43.11 | 46.02 | 46.42 | | GRPO | 46.89 | 48.55 | 49.45 |

Blank notes help (prompt-structure effect), but abstractions beat blank notes (content effect). The unique value of abstractions holds in both modes.

Cross-domain transfer

MATH-derived abstractions (Table 3):

| Test set | Baseline | GRPO | GRPO_blank | GRPO_train | |----------|----------|------|------------|------------| | GSM8K | 54.82 | 77.34 | 81.30 | 80.48 | | OlympiadBench | 13.91 | 16.04 | 16.36 | 17.34 | | MuSR-MATH | 40.94 | 47.06 | 45.21 | 46.20 | | MuSR-FOLIO | 44.58 | 44.82 | 45.54 | 45.54 |

Abstractions transfer to harder Olympiad problems and MuSR logical reasoning — they capture transferable strategies, not memorized problem types. Notably, on easy GSM8K, GRPO_train (80.48) trails GRPO_blank (81.30): abstractions help hard problems, hurt easy ones.

Difficulty stratification: abstraction as perturbation

Splitting MATH-500 by pre-injection accuracy (Table 6, average pass@1 change):

| Accuracy bucket | Baseline | GRPO | GRPO_train | |-----------------|----------|------|------------| | 0–50% (hard) | +6.31 | +3.62 | +0.83 | | 50–100% (mid) | +2.46 | -5.22 | -3.61 | | 100% (easy) | -2.32 | -5.29 | -1.12 |

Explanation: abstraction perturbs the output distribution. Base models have a diffuse distribution; perturbation widens the search and helps. RL-trained models have a sharpened distribution; perturbation pushes them off the optimum. Like humans: mistake notebooks help beginners, distract experts.

Cross-model transfer: can notes be lent to other models?

Table 4, injecting Llama-3.2-3B's abstractions elsewhere:

| Target model | Baseline | Inference_blank | Inference_abs | |--------------|----------|-----------------|---------------| | gemma-2-2B-it | 21.77 | 13.08 | 13.75 | | gemma-3-3B-it | 44.88 | 42.02 | 40.15 | | Phi-3.5-mini-instruct | 40.62 | 42.30 | 43.60 |

Mixed results: Phi-3.5 gains (+3.0), both Gemma models suffer. Cross-model transfer is model-specific. Since even blank notes hurt Gemma, it is likely sensitive to prompt-format changes rather than to the abstractions themselves.

Failure-mode analysis for Qwen

Why does Qwen dip at inference time (-1.09)? The paper rules out one hypothesis and confirms two culprits:

Ruled out: caution notes over-constraining reasoning. Caution count doesn't correlate with score drops (Pearson ρ=+0.069, p=0.13); more cautions actually correlate with *smaller* drops. Cautions are not negative constraints.
Culprit 1: format sensitivity. Missing-oxed{} rate rose from 3.8% to 4.6%; on the worst-hit problem, 24/32 samples computed the right answer but botched the boxed format (baseline: 11/32). Right reasoning, wrong formatting.
Culprit 2: poor retrieval fit. Sometimes retrieved abstractions are irrelevant (e.g., a greatest-common-divisor note retrieved for a minimum-sum problem) and mislead the model.

Both are engineering problems of abstraction injection, not flaws of abstraction itself.

Honest limitations

1. Tested only on math and logical reasoning; effects on open-ended generation, code, dialogue unknown. 2. Fixed design choices (teacher, encoder, k=6, threshold 0.85) with no sensitivity analysis. 3. Cross-model transfer is limited and model-specific. 4. Limited compute budget: one RL round, 8 rollouts per problem. 5. Degradation on easy problems — when to inject abstractions is itself an unsolved problem.

Deeper implications: experience abstraction as a self-improvement signal

Current LLM self-improvement has two main routes: self-training (learning only from solved problems) and reflection (test-time compute, nothing deposited into weights). Notes to Self takes a third path: distill abstract experience from both successes and failures, store it, retrieve it, and internalize it via RL. This loops at the *strategy* level, not the *problem* level:

1. Transferable: "try substitution for quadratics under a root" works across problems and domains, as the OlympiadBench/MuSR results show. 2. Auditable: abstractions are human-readable natural language — you can open the model's mistake notebook and inspect what it learned.

That self-extraction rivals teacher extraction also implies small models already have a form of metacognition: they can identify reusable strategies and pitfalls from their own reasoning, without being specifically trained to do so — a capacity plausibly absorbed from human "learning from experience" narratives in pretraining data.

From teacher-dependent to teacher-free

If self-extraction matches teacher extraction, self-improvement no longer depends on a strong teacher: a model can generate trajectories, distill abstractions, run RL on them, and deploy — with only outcome rewards in the loop. Open questions remain: library growth and retrieval efficiency, potential degradation from learning from one's own possibly-wrong notes, and difficulty-aware injection. But as a proof of concept, Notes to Self shows experience abstraction is a viable self-improvement signal that small models can produce themselves.

Writing notes to itself is a hallmark of human learning. Now AI does it too.

---

Paper: https://arxiv.org/abs/2607.20372 HTML version: https://arxiv.org/html/2607.20372v1 Code: https://github.com/ChangLiu-DrPatient/Notes-to-self