LSE: A Learning Self-Evolution RL Framework That Trains LLMs to Improve Their Own Prompts

Forum topic · ✨步子哥 · 2026-04-14

Summary

LSE (Learning Self-Evolution) is a reinforcement learning framework that converts the multi-step self-evolution of large language models into a single-step contextual bandit objective. A small policy model (Qwen3-4B-Instruct) is explicitly trained to revise prompts based on performance summaries, using an improvement-based (delta) reward defined as the difference in average execution accuracy on a fixed 5-10 sample validation set before and after an edit. At test time, an evolution tree is maintained and expanded via a UCB-based select-expand-evaluate loop, enabling recovery from bad edits through backtracking. Experiments on the BIRD Text-to-SQL benchmark show a 4B model reaching 67.3% execution accuracy, surpassing Claude 3.5 Sonnet (64.5%) and GPT-5 self-evolution (65.2%). The learned evolution skill transfers across models, boosting Arctic-7B by 6.7% without additional training, and can serve as a prompt optimizer for closed-source models. Ablations confirm that delta rewards outperform absolute rewards and that UCB tree search is more robust than linear-chain evolution.

1. Single-Step Reinforcement Learning Objective

Core idea

After deployment, LLMs face a "static factory-settings" bottleneck: capabilities are frozen in parameters and cannot adapt from interaction experience. Existing self-improvement methods (Reflexion, TextGrad, etc.) rely entirely on the model's inherent reasoning ability and never explicitly train the skill of "how to revise instructions based on failure cases." Such implicit reasoning must simultaneously perform credit assignment, gradient estimation, and exploration-exploitation balancing—hard to do reliably with natural-language reasoning alone.

LSE fundamentally simplifies multi-step self-evolution into a single-step RL objective. The original cumulative objective $\max_{f_\psi} \sum_{t=0}^{T} \gamma^t \bar{R}(c_t)$ with $c_{t+1} = f_\psi(c_t, S_t)$ suffers from severe long-horizon credit assignment. LSE compresses the horizon to T=1, adopting a contextual bandit framework so each edit decision gets immediate, explicit feedback.

Math formulation

The evolution policy $f_\psi$ takes the current context $$c_0$$ and a performance summary $$S_0$$ (validation accuracy, error-pattern analysis, representative failures), outputs $c_1 \sim f_\psi(\cdot | c_0, S_0)$ , and immediately receives reward:

r_{\text{LSE}} = \bar{R}(c_1) - \bar{R}(c_0)

where $\bar{R}(c)$ is the average downstream performance on a fixed validation set $$D$$ . This improvement-based reward is incentive-compatible: improvement is possible even from a strong starting point, and any degrading edit receives negative feedback—naturally encouraging continued exploration and avoiding premature convergence.

Training

Architecture: Qwen3-4B-Instruct as the policy backbone; the downstream action model $\pi_\theta$ is frozen, decoupling evolution skill from the task model.
Data: dynamically generated $$(c_0, S_0, c_1, r)$$ tuples via environment interaction. $$D$$ has 5-10 samples, each evaluated over 8 generations for reliability; $$D$$ is fixed so rewards stay comparable.
Optimization: policy gradients with baseline simplification (since $\bar{R}(c_0)$ is constant given $$c_0$$ ). Learning rate $1 \times 10^{-5}$ , 32 nodes per batch, 4 rollouts per node, 4 epochs. Compatible with PPO/GRPO by swapping in the delta reward.

2. Tree-Guided Search at Test Time

Evolution tree

Each node stores $(c_n, S_n, \bar{R}_n, v_n)$ : context, performance summary, average reward, and visit count. The tree serves both as a search structure and a knowledge accumulator, enabling recovery from failed explorations via backtracking to high-performing ancestors.

UCB selection

n^* = \arg\max_{n \in G} \left( \bar{R}_n + C \sqrt{\frac{\ln N}{v_n}} \right)

The exploitation term $\bar{R}_n$ favors historically good nodes; the exploration term rewards under-visited nodes. $$C=0$$ reduces to greedy; moderate values (e.g., $$C=2$$ ) work best. UCB's logarithmic regret bound provides theoretical grounding.

Tree vs. linear chain

| Property | Linear chain | UCB tree search | |:---|:---|:---| | Error recovery | None—errors accumulate | Fast backtracking recovery | | Exploration | Sequential local search | Globally balanced | | Guarantee | None | Asymptotically optimal |

On the BIRD Card Games dataset, a linear chain collapsed from ~60% to ~20% after one bad edit and never recovered; UCB tree search backtracked and returned to ~60% within a few rounds, eventually converging higher.

Test-time loop

1. Initialize root node with the seed prompt and its validation metrics. 2. Iterate: UCB-select a node → generate a child context with $f_\psi$ → evaluate on $$D$$ and update node info. 3. Output $c_{\text{best}} = \arg\max_{n \in G} \bar{R}_n$ —not necessarily the last-created node.

3. Delta-Based Reward Design

Why absolute rewards fail

Learning suppression at high baselines: once a high-scoring context (~90%) is found, most edits score lower, causing premature convergence.
Task-difficulty bias: seed-prompt baselines on BIRD range from 52.3% (Formula 1) to 65.3% (Codebase); absolute rewards incentivize picking easy tasks.
Conservative convergence: absolute-reward variants plateau, while delta-reward variants keep improving.

Delta reward advantages

| Dimension | Mechanism | Effect | |:---|:---|:---| | Difficulty-invariant | Subtracting $\bar{R}(c_0)$ normalizes difficulty | +5% on hard == +5% on easy | | Sustained exploration | No penalty for falling from a height | Structural edits are encouraged | | Single-step fit | Two-state comparison | A 4B model can learn it |

For Text-to-SQL, correctness uses execution accuracy, not string matching. $\bar{R}(c)$ averages correctness over $$D$$ ; statistical significance checks can gate positive rewards against noise.

Ablation results

$A_{\text{GRPO}}$ (absolute): rapid early rise, plateaus at ~62%.
$A_{\text{LSE}}$ (delta): continuous improvement, reaching ~67%.

Cross-model transfer

Evolution instructions trained on Qwen3-4B transfer to Arctic-7B for a +6.7% gain (57.7% → 64.4%) with no additional training. The mechanism is decoupling of the meta-skill ("how to improve prompts from feedback") from task content. A 4B LSE model can even act as a prompt optimizer for closed-source models, bringing GPT-5's self-evolution performance to 65.2%, close to LSE's own 67.3%.

4. Experimental Results

BIRD Text-to-SQL benchmark

| Method | Model scale | Avg execution accuracy | vs. seed | |:---|:---|:---|:---| | Seed Prompt | — | 57.2% | — | | Claude 3.5 Sonnet | ~175B | 64.5% | +12.8% | | GPT-5 (self-evolution) | ~1.8T | 65.2% | +14.0% | | LSE (Qwen3-4B) | 4B | 67.3% | +17.7% |

Per-database gains: Financial +11.5% (56.8→68.3), Toxicology +7.8%, Codebase +6.2%, Formula 1 +4.7%, Card Games +3.5%.

Compared to static fine-tuning, LSE adaptation requires no extra labeled data, only validation-set evaluations: $O(|D| \times T \times \text{cost}_{\text{eval}})$ per database, typically 10-20 evolution rounds.

Marginal gains per round

| Rounds | Cumulative gain | Marginal/round | Phase | |:---|:---|:---|:---| | 0→5 | +8% | +1.6% | Fast capture of obvious improvements | | 5→10 | +3% | +0.6% | Fine-tuning | | 10→15 | +1% | +0.2% | Diminishing returns | | 15→20 | +0.5% | +0.1% | Near convergence |

25 rounds is the standard configuration; adaptive termination (stop after multiple no-improvement rounds) is a future direction.