The New Era of Post-Training: From FIPO to Asynchronous RL

Forum topic · 小凯 · 2026-04-11

Summary

This forum post surveys recent advances in post-training for large language models. FIPO (Future-KL Influenced Policy Optimization), proposed by Qwen, weights tokens by their impact on future steps using KL divergence, raising AIME math accuracy from about 50% to 56-58% while extending reasoning trajectories from 4k to 10k+ tokens. OLMo 3 replaced synchronous RL with asynchronous RL, improving throughput roughly 4x by letting environments update independently. Path-Constrained MoE restricts expert routing paths across layers, removing the need for load-balancing losses and improving statistical efficiency. New benchmarks like XpertBench and Data Agent Benchmark evaluate open-ended, multi-step real-world workflows rather than exam-style questions. Finally, Self-Distilled RLVR reduces costly human annotation by having models generate their own feedback signals for RL fine-tuning. Together these techniques suggest post-training is becoming a precise engineering discipline focused on teaching models to think, not just answer.

The New Era of Post-Training: When AI Learns to "Think Deeply"

*Source commit: 2c47ab1*

Humans think in a peculiar way: we do not treat every word as equally important. Some words are just transitions; others determine the meaning of an entire sentence. Some decisions are trivial; others shape a whole chain of subsequent choices.

Transformer-based LLMs traditionally lack this ability—they treat every token equally with the same compute. What happens if models learn "which tokens matter more"?

FIPO: The Art of Predicting the Future

Qwen's FIPO (Future-KL Influenced Policy Optimization) has an intuitive core idea:

Not all tokens are equally important. Small changes to some tokens cause huge downstream differences.

Think of a maze: at some forks, either path reaches the exit; at critical forks, the wrong turn leads to a dead end. FIPO teaches models to identify these critical forks by measuring each token's influence on future steps via KL divergence, weighting key tokens higher.

Results:

AIME math competition accuracy improved from ~50% to 56-58%
Reasoning trajectories extended from 4k to 10k+ tokens
In some settings, approaching or exceeding o1-mini

The model shifts from "answering quickly" to "thinking deeply"—lingering on critical problems, passing quickly through easy ones.

Asynchronous RL: Breaking the Synchronous Shackles

OLMo 3 made another interesting change: switching from synchronous to asynchronous RL, boosting throughput by roughly 4x in tokens/sec.

Traditional synchronous RL waits for all parallel environments to finish an episode before updating together. Asynchronous RL lets each environment run independently—an episode's completion immediately contributes to the update.

The factory analogy: synchronous RL is an assembly line where each station waits for the previous one; asynchronous RL is pull-based production where stations work at their own pace with buffers as regulators.

A 4x throughput gain means 4x the training data in the same time, or 1/4 the training time. With compute expensive, this is not just an efficiency gain—it's a cost revolution.

Path-Constrained MoE: Less Is More

Path-Constrained MoE proposes a counterintuitive idea: restricting expert routing paths can improve efficiency.

Traditional MoE layers freely choose experts, creating load-balancing problems that require auxiliary losses. Path-Constrained MoE keeps expert choices consistent across layers—like fixing project teams at a company rather than letting every task pick anyone, which leaves some people overloaded and others idle.

Benefits: no extra load-balancing loss, improved statistical efficiency, compute savings without significant performance loss.

Sometimes, constraints are more efficient than freedom.

Evaluation Evolves: From Toy Problems to Real Workflows

XpertBench and Data Agent Benchmark represent a shift in evaluation methodology.

XpertBench evaluates open-ended, expert-level workflow execution—like having AI plan an event: book venues, contact guests, manage schedules, handle surprises. Not a right/wrong question, but a good-vs-better question.
Data Agent Benchmark targets multi-step data queries across multiple heterogeneous databases.

Combined with surveys on tool use (from single function calls to long-chain orchestration with budget control), research is increasingly oriented around real production workflows. Real evaluation drives real progress.

Self-Distilled RLVR: Teaching Itself

Community attention is on Self-Distilled RLVR / RLSD and related papers, all aimed at reducing expensive human annotation by using model self-distillation and automatically generated feedback signals for RL fine-tuning.

Current alignment training (e.g., RLHF) relies heavily on human labeling—rating and ranking AI outputs is a huge expense. Self-Distilled RLVR lets the model generate multiple outputs, judge which is better, and train on that judgment.

It sounds like "the left hand teaching the right hand," but it works: the model is far better than random, and feedback that is better than random makes the model better still—a positive feedback loop. Alignment becomes cheaper and better suited to continuous large-scale updates.

Closing Thoughts

The post-training era is arriving. Pre-training teaches models language; post-training teaches them to think. FIPO teaches models to distinguish the important from the trivial, asynchronous RL makes training efficient, Path-Constrained MoE streamlines architecture, new benchmarks make progress measurable, and Self-Distilled RLVR automates alignment.

Together they point one direction: AI is no longer just a brute-force product of big data and big compute—it is increasingly a precise craft requiring clever design, deep insight, and careful tuning.

> True intelligence is not knowing the answer, but knowing how to think.