Bidirectional Evolutionary Search: When Language Models Learn to Self-Improve

Forum topic · 小凯 · 2026-05-28

Summary

This forum post introduces a paper titled 'Self-Improving Language Models with Bidirectional Evolutionary Search' (arXiv:2605.28814) by Guowei Xu, Zhenting Qi, Huangyuan Su, Weirui Ye, Himabindu Lakkaraju, Sham M. Kakade, and Yilun Du. The proposed method, Bidirectional Evolutionary Search (BES), addresses two limitations of existing self-improvement approaches: sparse feedback from best-of-N sampling and tree search, and the inability of autoregressive generation to recombine promising partial solutions. BES combines a forward search with evolutionary operators that recombine existing trajectory segments, escaping what the authors call the 'entropy shell' of low-diversity candidates, with a backward search that recursively decomposes a target into verifiable subgoals, converting sparse final-answer feedback into dense intermediate feedback. The paper provides theoretical guarantees that evolutionary operators can escape the entropy shell and that backward decomposition can exponentially reduce the sample complexity of finding correct solutions. Experiments show BES achieves sustained post-training improvements where mainstream algorithms stall, and outperforms existing open-source frameworks on math reasoning, code generation, and scientific question-answering benchmarks. An ablation study confirms the bidirectional combination significantly outperforms either direction alone.

AI's Evolutionary Theory: When Language Models Learn to "Self-Breed"

> arXiv: 2605.28814 | Self-Improving Language Models with Bidirectional Evolutionary Search > Authors: Guowei Xu, Zhenting Qi, Huangyuan Su, Weirui Ye, Himabindu Lakkaraju, Sham M. Kakade, Yilun Du > Published: 2026-05-27

Introduction: When AI Starts Teaching Itself

Imagine a child learning math. The teacher gives him a problem and he gets it wrong. The teacher says: "Wrong, think again." He tries again and is wrong again. The teacher only says: "Wrong."

This is the dilemma facing current AI training methods. Most "self-improvement" approaches, like best-of-N sampling (picking the best of N candidates), act like a teacher who only says "wrong" — they tell you the answer is bad, but not *why* it's bad or *how* to improve it.

The paper proposes Bidirectional Evolutionary Search (BES), which works like a super coach who can decompose problems and give concrete advice.

Chapter 1: Two Blind Spots of Evolution

1.1 The Dilemma of Sparse Feedback

Two mainstream search approaches have limitations:

Best-of-N sampling: The model generates N candidate answers, a verifier scores them, and the highest-scoring one is selected. The problem: feedback is sparse. The verifier only scores the final answer, ignoring intermediate steps — like a teacher who checks only the final answer, never your scratch work.

Tree search (e.g., MCTS): Better in that it considers intermediate states, but the core issue remains: the evaluation signal is sparse — node evaluations are typically based only on "probability of winning from this node," not on "what problems this node itself has."

1.2 The Autoregressive Cage

The second blind spot is subtler: candidate construction is limited by autoregressive generation.

Problem 1: Exploration is confined to high-probability regions. Models tend to generate "familiar" content; if a correct answer requires an expression the model is not good at, autoregressive generation struggles to discover it.

Problem 2: Existing parts cannot be recombined. If candidate A has a good first half and candidate B has a good second half, the ideal improvement would combine them — but autoregressive generation must regenerate from scratch.

The paper states this precisely: candidates are confined to a narrow "entropy shell" — they share similar entropy and cluster in the model's comfort zone, lacking diversity.

Chapter 2: Bidirectional Evolutionary Search

2.1 Forward Search: Evolution, Not Just Growth

BES's forward search does two things:

Expansion: The model generates next steps from the current state — "growth."

Evolution operators: BES's uniqueness — it recombines existing partial trajectories. Like Lego: instead of building step by step from instructions, you can disassemble and recombine parts of two half-finished models. Benefits: 1. Escaping the entropy shell: combinations produce novel candidates unreachable by any single path 2. Leveraging prior exploration: improvements build on existing successes rather than starting over

The paper provides a theoretical guarantee: evolution operators allow candidates to escape the narrow entropy shell — expansion-only candidates are provably confined to a low-entropy region, while evolution can jump to higher-entropy regions where better solutions may exist.

2.2 Backward Search: Decomposing Big Problems into Small Games

BES's more radical innovation: decomposing the goal from the "endpoint" backward into verifiable subgoals. For a task like "write a good essay on climate change":

Main goal: write a good essay
Subgoal 1: good introduction (checkable: does it cover the research question?)
Subgoal 2: good background (checkable: are key references cited?)
Subgoal 3: good argumentation (checkable: is the logic sound?)
Subgoal 4: good conclusion (checkable: does it answer the question?)

Each subgoal is checkable — the verifier can independently judge whether it's achieved. This converts sparse "final feedback" into dense "intermediate feedback."

2.3 Recursive Decomposition

Decomposition is recursive: if "write a good argument" is still too complex, it's broken down further (find supporting evidence → build rebuttal structure → write logical connections). Like peeling an onion, a mountain-sized task becomes a pile of pebbles, with checkpoints at every level.

2.4 The Magic of Coupling

Forward and backward searches are coupled:

Backward decomposition guides forward exploration: knowing the subgoals gives search direction.
Forward results correct backward decomposition: if a subgoal turns out unreachable, the decomposition strategy is adjusted.

Like an architect and construction crew collaborating: the architect draws blueprints, the crew builds, and problems discovered on-site feed back into revised blueprints.

Chapter 3: Theoretical Guarantees

3.1 The Entropy Shell, Mathematically

Imagine candidate space as a giant sphere where each candidate's "entropy" is its distance from the center. Autoregressive candidates cluster in one shell of the sphere. Evolution operators (recombination) allow "jumps" across shells and regions, breaking the entropy shell constraint.

3.2 Exponentially Reduced Sample Requirements

If a problem needs 10 steps with 2 choices each, random search needs 2^10 = 1024 attempts. But if backward search decomposes it into 10 independently verifiable subgoals:

Feedback arrives after each correct step, not only after the full 10-step path
The exponential search space compresses toward linear

The paper rigorously proves that backward search can exponentially reduce the number of samples needed to find a correct answer.

Chapter 4: Experiments

4.1 Post-Training Breakthrough

On post-training improvement, mainstream post-training algorithms all failed (models could not improve consistently), while BES achieved sustained improvement — because it provides dense intermediate feedback via bidirectional search rather than sparse final-answer signals.

4.2 Inference-Time Dominance

BES was tested on three open-ended problem-solving benchmarks: 1. Mathematical reasoning 2. Code generation 3. Scientific question solving

BES outperformed all existing open-source frameworks on both "average performance" and "best performance" metrics — indicating both stability and exploration capability.

4.3 Bidirectional Advantage: 1+1 > 2

Ablation results:

Forward only: better than baseline, worse than BES
Backward only: better than baseline, worse than BES
Bidirectional: significantly better than either alone

Chapter 5: Why BES Is an Important Step for AI Evolution

From "trial and error" to "evolution": BES combines mutation (expansion), recombination (evolution operators), and selection pressure (subgoal verification) — mirroring biological evolution's mutation, recombination, and natural selection.
From "black-box verification" to "white-box guidance": the verifier no longer just scores final results; it checks every subgoal, giving interpretable, actionable feedback.
From "unidirectional growth" to "bidirectional optimization": humans solve problems by reasoning forward from the present, backward from the goal, and adjusting midway. BES is the first to implement this bidirectional, iterative, adaptive optimization in AI search.

Conclusion: The Age of AI Self-Breeding

Darwin identified three elements of evolution: mutation, selection, and heredity. BES implements analogous mechanisms:

Expansion → mutation (new candidates)
Evolution operators → recombination (combining strengths)
Backward search → selection pressure (subgoal verification)
Forward search → heredity (retaining successful paths)

But BES is faster and more goal-directed than biological evolution: generations complete in seconds, and backward search provides direction that natural evolution lacks.

> *"Evolution is not about the strongest surviving, but about those most sensitive to change surviving. BES makes AI sensitive to change — and to itself."*

Reference

Xu, G., Qi, Z., Su, H., Ye, W., Lakkaraju, H., Kakade, S. M., & Du, Y. (2026). Self-Improving Language Models with Bidirectional Evolutionary Search. *arXiv preprint arXiv:2605.28814*.