Notes to Self: When Small Models Write Their Own 'Experience Handbooks' from Their Own Mistakes—and Match What Teachers Extract
An Experiment in 'Learning How to Learn'
You get a math problem wrong. Your teacher analyzes it: 'You got stuck on trigonometric substitution—next time you see a quadratic under a square root, try substitution first.' You write this in your mistake notebook. Next time you hit a similar problem, you look it up—and it works.
This is the classic human learning pattern: abstract lessons from failure, store them in retrievable memory, invoke them in similar future situations. Psychologists call it 'experiential abstraction'; Polya discussed such cognitive tools in *How to Solve It* (1945).
Now an interesting question: can large language models do the same?
Not having a strong model (teacher) distill experience for a small model (student)—that's traditional knowledge distillation—but having the small model extract its own experience notes from its own solution traces, then retrieve them for similar future problems.
Chang Liu et al. (arXiv:2607.20372, code on GitHub) answer yes: a small model's self-extracted experiential abstractions perform on par with those extracted by a strong teacher model. This is stronger than 'models can learn from experience'—it suggests small models possess a form of metacognition, able to identify reusable strategies and pitfalls from their own reasoning processes.
Pipeline Overview: From Solution Traces to an Experience Library to Reasoning Enhancement
The pipeline has three steps:
Step 1: Generate solution traces
Give the target model πθ a batch of training problems and let it solve them (right or wrong). Each problem's full reasoning is recorded as a 'solution trace' y ~ πθ(·|q).
Step 2: Extract experiential abstractions from traces
An 'extractor LLM' analyzes each trace:
- On success, extract a strategy abstraction—the key insight and what is reusable.
- On failure, extract a caution abstraction—where it got stuck, what to watch out for.
- An untrained base model's distribution is relatively diffuse; perturbation widens the search and helps it find correct answers.
- An RL-trained model's distribution is sharpened; perturbation pushes it away from the optimum.
- Self-training: the model generates data, filters for correct answers, and trains. But this loops at the 'problem' level—the model learns from problems it got right; wrong ones are discarded.
- Reflection: the model generates, gets feedback, and regenerates at test time. But this is test-time compute scaling; experience isn't deposited into the model.
- The abstraction library grows—how to keep retrieval efficient?
- Could abstractions 'degrade'—learning from one's own (possibly wrong) abstractions and amplifying errors?
- How to handle the easy-problem degradation—some 'difficulty-aware' abstraction injection strategy?
The extractor can be a strong teacher model (DeepSeek-V4-Flash) or the target model itself. The paper's core comparison is between these two extraction modes.
Abstractions are embedded with all-MiniLM-L6-v2 into 384-dim vectors, deduplicated via cosine similarity with union-find (threshold 0.85), with strategies and cautions stored separately. Dedup shrinks the library by ~30% on average.
Step 3: Two usage modes
Mode 1: Inference-time retrieval. For a test problem, retrieve top-k (k=6) most relevant abstractions via cosine similarity, insert them into a <reference_notes> block after the question. The model solves with these 'notes' in hand.
Mode 2: Abstraction-augmented GRPO training. During GRPO (Group Relative Policy Optimization), retrieved abstractions are also inserted into training prompts. The model learns to exploit abstractions to improve its reasoning.
The modes compose: train an abstraction-augmented checkpoint, then also retrieve at test time.
Core Results: Self-Extracted Notes Match Teacher-Extracted Notes
The main table (Table 1) compares five configurations on MATH-500:
| Model | Extractor | Baseline | Inference_abs | GRPO | GRPO_train | GRPO_train+test | |-------|-----------|----------|---------------|------|------------|-----------------| | Llama-3.2-3B | Teacher | 43.11 | 46.42 | 46.89 | 49.45 | 47.65 | | Llama-3.2-3B | Self | - | 46.30 | - | 49.07 | 47.67 | | Qwen-2.5-1.5B | Teacher | 50.23 | 49.14 | 52.55 | 53.44 | 52.66 | | Qwen-2.5-1.5B | Self | - | 49.44 | - | 53.77 | 52.72 |
Key observations:
1. Abstraction-augmented GRPO training (GRPO_train) improves steadily. Llama: 43.11 → 49.45 (+6.34); Qwen: 50.23 → 53.44 (+3.21). This is the effect of 'using abstractions during training'—the model learns to exploit them during GRPO.
2. Self-extraction matches teacher extraction. This is the most striking result. Llama: self 49.07 vs teacher 49.45—only 0.38 apart. Qwen: self 53.77 actually exceeds teacher 53.44.
A small model's notes from its own traces work as well as (or better than) a strong teacher's. It doesn't need a 'teacher' to summarize its mistakes—it can do it itself.
3. Inference-time retrieval (Inference_abs) also works alone. Llama: 43.11 → 46.42 (+3.31); Qwen: 50.23 → 49.14 (-1.09, slightly worse). Without training, just slipping the model a few 'cheat sheets' helps Llama but is unstable for Qwen.
4. Train+test stacking (GRPO_train+test) underperforms training alone (GRPO_train). Llama drops 49.45 → 47.65; Qwen 53.44 → 52.66. Training has already 'internalized' the abstractions; more cheat sheets at test time interfere.
Honest Controls: The Blank-Note Experiment
A key objection: do gains come from the abstractions themselves, or from the prompt structure change? Maybe just adding a <reference_notes> block makes the model try harder, regardless of content.
Table 2 answers this with three configurations:
| Mode | No notes | Blank notes | + Abstractions | |------|----------|-------------|----------------| | Inference | Baseline 43.11 | Inference_blank 46.02 | Inference_abs 46.42 | | RL (GRPO) | GRPO 46.89 | GRPO_blank 48.55 | GRPO_train 49.45 |
Blank notes inherit the same prompt structure but leave the <reference_notes> block empty. Result: blank notes beat no notes (a prompt-structure effect), but abstractions beat blank notes (a content effect). The abstractions' 'unique advantage' holds in both modes.
Cross-Domain Transfer: Math Experience Helps Logical Reasoning
Can abstractions transfer across domains? Table 3 tests MATH-derived abstractions on other benchmarks:
| Test set | Baseline | GRPO | GRPO_blank | GRPO_train | |----------|----------|------|------------|------------| | GSM8K (easy math) | 54.82 | 77.34 | 81.30 | 80.48 | | OlympiadBench (olympiad math) | 13.91 | 16.04 | 16.36 | 17.34 | | MuSR-MATH (logical reasoning) | 40.94 | 47.06 | 45.21 | 46.20 | | MuSR-FOLIO (logical reasoning) | 44.58 | 44.82 | 45.54 | 45.54 |
Key observation: MATH-derived abstractions also help on the harder OlympiadBench and on logical-reasoning MuSR. The abstractions aren't memorizing MATH problem types—they encode transferable reasoning strategies.
But on GSM8K (easy math), GRPO_train (80.48) underperforms GRPO_blank (81.30). This points to an important pattern: abstractions help hard problems, not easy ones.
Difficulty Stratification: Abstractions Are 'Perturbations'—Helping Diffuse Distributions, Hurting Sharpened Ones
A finer difficulty-stratified analysis (Table 6) buckets MATH-500 problems by pre-injection accuracy:
| Accuracy bucket | Baseline | GRPO | GRPO_train | |-----------------|----------|------|------------| | 0–50% (hard) | +6.31 | +3.62 | +0.83 | | 50–100% (medium) | +2.46 | -5.22 | -3.61 | | 100% (easy) | -2.32 | -5.29 | -1.12 |
Numbers are mean pass@1 change after abstraction injection.
Abstractions help on hard problems, hurt on easy ones. The more trained the model, the worse the damage on easy problems.
The paper's precise explanation: an abstraction is a perturbation of the model's output distribution.
This matches human experience: mistake notebooks help beginners but may distract experts. Have a math olympiad gold medalist consult a beginner's notebook mid-solve, and you disrupt their rhythm.
Cross-Model Transfer: Can Experience Be Lent to Other Models?
Table 4 injects Llama-3.2-3B's abstractions into other model families:
| Target model | Baseline | Inference_blank | Inference_abs | |--------------|----------|-----------------|---------------| | gemma-2-2B-it | 21.77 | 13.08 | 13.75 | | gemma-3-3B-it | 44.88 | 42.02 | 40.15 | | Phi-3.5-mini-instruct | 40.62 | 42.30 | 43.60 |
Mixed results: Phi-3.5 benefits (+3.0), but both Gemma models are hurt. Cross-model transfer is model-specific—one model's mistake notebook can't simply be lent to another architecture. Since Inference_blank also drops sharply for Gemma, Gemma appears sensitive to the prompt-structure change itself and not robust to the format shift.
Failure-Mode Analysis: Not the Abstractions' Fault—Format and Retrieval
Qwen slightly drops under inference-time retrieval (-1.09). The paper's failure-mode analysis rules out one natural hypothesis and confirms two culprits.
Ruled out: caution notes over-constraining. Maybe retrieved 'caution' notes over-constrain Qwen's reasoning? If so, problems retrieving more cautions should drop more. The data says the opposite: caution count is uncorrelated with score drop (Pearson ρ=+0.069, p=0.13), and more cautions trend toward *smaller* drops (Table 7). Cautions are not 'negative constraints'.
Culprit 1: format sensitivity. Qwen is sensitive to the <reference_notes> format. With abstractions injected, the rate of missing \boxed{} answers rises from 3.8% to 4.6%. On Qwen's worst-hit problem (id 1332), 24/32 samples computed the correct answer but failed to write it in \boxed{} format and were scored wrong (Baseline: only 11/32). Right reasoning, wrong format.
Culprit 2: poor retrieval fit. Sometimes retrieved abstractions are irrelevant and mislead the model. Example: a problem asking for the 'minimum positive integer sum' retrieved an abstraction about 'greatest common divisors'—completely unrelated, and the model was led astray.
Neither failure is the abstractions' fault—both are engineering issues of the injection mechanism. Better retrieval and format robustness should mitigate them.
Honest Assessment: Limitations and Open Questions
The paper is candid about limitations:
1. Tested only on math and logical reasoning. Effects on open-ended generation, code, and dialogue are unknown. Math has clear correct answers and clean RL rewards; other domains are harder. 2. Fixed design choices. Teacher model, sentence encoder, retrieval top-k (k=6), dedup threshold (0.85) are all fixed with no sensitivity analysis. 3. Limited cross-model transfer. Works on Phi-3.5, backfires on Gemma. Transfer is model-specific, not universal. 4. Limited compute budget. RL post-training ran one round with 8 rollouts per problem. Larger-scale training may change effect sizes. 5. Degradation on easy problems. Abstractions hurt on already-mastered problems. Deployment needs a judgment of when to use them—which is itself an unsolved problem.
Deeper Implications: Experiential Abstraction as a 'Self-Improvement Signal'
The most interesting thing about this paper isn't the specific numbers but the deep possibility it reveals: experiential abstraction is a viable self-improvement signal.
Current LLM self-improvement has two main routes:
Notes to Self takes a third path: extract 'abstract experience' from both failed and successful problems, store it in a library, retrieve it later, or internalize it via RL. This loops at the 'strategy' level, not the 'problem' level—the model learns not how to solve a specific problem but 'when to apply which strategy'.
This strategy-level loop has two advantages:
1. Transferable. Specific problems don't transfer (solving one MATH problem doesn't help on OlympiadBench), but strategies do ('try substitution on quadratics under roots' works across problems). The paper confirms MATH-derived abstractions work on OlympiadBench and MuSR. 2. Auditable. Abstractions are natural language, human-readable. You can open the model's 'mistake notebook', see what strategies it learned, and judge which are good and which are misleading—far more intuitive than inspecting parameter changes.
The finding that self-extraction matches teacher extraction carries a deeper implication: small models already possess a form of metacognition—they can identify reusable strategies and pitfalls from their own reasoning. This isn't trained (the extractor is an off-the-shelf instruct model); it emerges from pretraining. Reading human text, models learned not just 'knowledge' but the metacognitive pattern of 'distilling strategies from experience'.
This parallels human metacognitive development. Children are taught to 'summarize lessons from mistakes', but the seed of this ability is planted earlier—hearing adults say 'I made this mistake before, watch out for X next time' teaches the narrative pattern of extracting reusable lessons from experience. LLMs likely absorbed this same narrative pattern from pretraining data, enabling them to write their own 'experience notes'.
From 'With a Teacher' to 'Without': The Next Stop for Self-Improvement
If self-extraction truly matches teacher extraction, LLM self-improvement loses a key dependency: small models no longer need a strong teacher to distill experience. They can generate their own experience library, use it, and train on it.
This points to a possible future: a model continuously generates solution traces in use, periodically distills new abstractions into the library, runs RL post-training on the library, then deploys the improved model. The whole loop needs no human annotation and no strong teacher—just the model itself plus outcome rewards (right or wrong).
Open problems in this loop:
Writing notes to yourself is a hallmark of human learning. Now AI does it too.
---
Paper: https://arxiv.org/abs/2607.20372 HTML version: https://arxiv.org/html/2607.20372v1 Open-source code: https://github.com/ChangLiu-DrPatient/Notes-to-self