The New Era of Post-Training: From FIPO to Asynchronous RL

Forum topic · 小凯 · 2026-04-11

Summary

A Chinese tech forum post surveys recent advances in post-training for large language models. FIPO (Future-KL Influenced Policy Optimization), proposed for Qwen models, weights tokens by their influence on future KL divergence, teaching models to identify critical decision points; it reportedly lifted AIME math accuracy from ~50% to 56-58% and extended reasoning trajectories from 4k to 10k+ tokens, approaching o1-mini in some settings. OLMo 3 switched from synchronous to asynchronous RL, achieving roughly 4x throughput in tokens per second by letting environments update independently. Path-Constrained MoE restricts expert routing paths across layers, removing the need for load-balancing losses while improving efficiency. New benchmarks like XpertBench and Data Agent Benchmark evaluate open-ended, expert-level workflow execution and multi-step cross-database querying instead of static test questions. Self-Distilled RLVR/RLSD methods reduce costly human annotation by using model-generated feedback signals for RL fine-tuning. Together these trends suggest post-training is becoming a precise engineering craft focused on efficient, thoughtful reasoning.

This post (source commit: 2c47ab1) surveys recent developments in post-training for large language models, arguing that the field is entering a new era where AI learns to "think deliberately" rather than treat every token equally.

FIPO: Predicting the Future

FIPO (Future-KL Influenced Policy Optimization), proposed for Qwen models, is built on a simple idea: not all tokens matter equally. Some small token changes cause large downstream differences—like critical forks in a maze.

FIPO measures each token's influence on subsequent steps (via KL divergence) and gives "critical tokens" higher weight in policy optimization. Reported results:

AIME math competition accuracy improved from ~50% to 56-58%
Reasoning trajectories extended from 4k to 10k+ tokens
In some settings, performance approaches or exceeds o1-mini

Asynchronous RL: Breaking the Synchronous Bottleneck

OLMo 3 replaced synchronous RL with asynchronous RL, improving throughput by roughly 4x (tokens/sec).

In synchronous RL, training must wait for all parallel environments to finish an episode before updating. Asynchronous RL lets each environment run independently and contribute updates as soon as its episode completes—like switching a factory from rigid sequential flow to pull-based production. A 4x throughput gain means 4x more data in the same time, or training time cut to a quarter—a cost revolution when compute is expensive.

Path-Constrained MoE: Less Is More

Path-Constrained MoE takes a counterintuitive approach: restricting expert routing paths improves efficiency. Traditional MoE lets every layer freely pick experts, creating load-imbalance problems that require extra balancing losses. By keeping expert choices consistent across layers (like a fixed project team), Path-Constrained MoE eliminates the load-balancing loss, improves statistical efficiency, and saves compute without noticeably hurting performance.

Evaluation Evolves: From Toy Questions to Real Workflows

XpertBench: evaluates open-ended, expert-level workflow execution (e.g., planning an event end-to-end) rather than exam-style questions with single correct answers.
Data Agent Benchmark: tests multi-step data queries across multiple heterogeneous databases.
A survey on tool use covers the shift from single function calls to long-chain orchestration with budget control.

The post argues this is a healthy trend: real evaluations drive real research progress.

Self-Distilled RLVR: Teaching Itself

Community attention is turning to Self-Distilled RLVR / RLSD papers, which reduce expensive human annotation by having models generate outputs, judge which is better, and use that signal for RL fine-tuning. Even though the model's judgments are imperfect, feedback "better than random" creates a positive feedback loop, making alignment cheaper and suitable for large-scale continuous updates.

Closing Thoughts

The author frames these advances as a common direction: AI is no longer just a brute-force product of "big data + big compute," but an increasingly precise craft requiring clever design and careful tuning. Pre-training teaches a model language; post-training teaches it to think.

> "True intelligence is not knowing the answer, but knowing how to think."