Multiplex Thinking: When LLMs Learn to Stack Branches into a Single Token

Forum topic · ✨步子哥 · 2026-01-14

Summary

A zhichai.net forum post reviews the paper 'Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge' (Tang et al., arXiv:2601.08808). The method addresses a core limitation of Chain-of-Thought (CoT) reasoning: each step commits to one discrete token, making exploration long, expensive, and DFS-like, while deterministic continuous-token approaches (Soft Thinking) cannot support on-policy RL exploration. Multiplex Thinking samples K candidate tokens independently at each step, averages their one-hot vectors, and merges them into one continuous 'multiplex token' of the same sequence length as standard CoT. Because the representation is built from discrete samples, trajectory probabilities factorize explicitly, enabling GRPO training with verifiable rewards. Entropy analysis shows the per-step exploration space expands from |V| to |V|^K. On DeepSeek-R1-Distill-Qwen 1.5B/7B models across six math benchmarks (AIME 2024/2025, AMC 2023, MATH-500, Minerva Math, OlympiadBench), the method wins 11 of 12 Pass@1 settings and reaches ~55% vs ~40% for discrete RL at Pass@1024 on AIME 2025 (7B). It also generates shorter responses at higher accuracy, and K=2 suffices for most gains with diminishing returns beyond.

Background

This post reviews “Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge” (Tang et al., 2026, arXiv:2601.08808v1).

Standard Chain-of-Thought (CoT) forces the model to commit to one discrete token per step — “hard reasoning.” Human thinking is softer: multiple possible next moves coexist until one becomes clearly better. CoT works, but sequences get long, token cost rises, and exploration resembles depth-first search — costly for on-policy RL (e.g., GRPO). Continuous-token approaches like Soft Thinking (Zhang et al., 2025) average vocabulary embeddings into “concept tokens,” preserving distributions cheaply — but they are deterministic: the same context always produces the same token, so rollouts never diversify, which makes them incompatible with RL exploration.

Key points

Mechanism: At each step, sample K tokens independently from the LM head; average their one-hot vectors into $s_i=\frac{1}{K}\sum_j z_{i,j}$ , map back to embedding space (optionally reweighted by LM-head probabilities): $c_i = E^\top(s_i \odot w_i)$ . K=1 degenerates to discrete CoT; K→∞ approaches the true distribution.
Self-adaptive behavior: When the model is confident, all K samples agree and the token behaves like a normal discrete token. When uncertain, samples diverge and the multiplex token encodes multiple branches — with no increase in sequence length.
RL compatibility: Since the token is built from K independent discrete samples, trajectory log-probability factorizes as $\log \pi(c \mid e(q)) = \sum_i \sum_j \log \pi_\theta(k_{i,j}\mid e(q), c_{<i})$ , enabling standard policy-gradient objectives with verifiable rewards (RLVR).
Entropy expansion: The composite action's entropy is $K \times$ the per-step Shannon entropy, expanding the effective exploration space from $$|V|$$ to $$|V|^K$$ — more breadth-first, rather than committing to one path.

Experimental setup

Backbones: DeepSeek-R1-Distill-Qwen-1.5B and 7B
Training: GRPO, 300 steps, batch size 128, lr 1e-6, KL and entropy penalties = 0, max response 4096 tokens, 8 rollouts per problem, temperature 1.0
Benchmarks (6): AIME 2024, AIME 2025, AMC 2023, MATH-500, Minerva Math, OlympiadBench
Baselines: Discrete CoT, Stochastic Soft Thinking (Wu et al., 2025), Discrete RL (same GRPO)

Results

Pass@1 (7B): Multiplex Thinking wins 11 of 12 settings, e.g., AMC 2023: 50.7 vs 44.7 (Discrete RL) and 47.9 (Soft); MATH-500: 78.0 vs 74.1; OlympiadBench: 41.7 vs 38.0.
Pass@k up to 1024: On AIME 2025 (7B), Discrete RL plateaus around 40% while Multiplex Thinking climbs to ~55% — the multiplex representation preserves low-probability correct paths.
Width K: K=2 already gives large gains; improvements continue to K=6 with diminishing returns. Default is K=3.
Token efficiency: At the same 4k length budget, Multiplex reaches 40.5 average accuracy vs 35.8 for Discrete CoT-4k and even beats CoT-5k (39.6) — gains come from denser tokens, not longer thinking.
Entropy retention: Entropy drops less during RL training with K>1 (5.82–7.09% vs 9.44% for K=1), meaning exploration persists longer.
Ablations: Uniform averaging vs LM-head reweighting perform similarly — the gain comes from the branch-and-merge paradigm itself. Qualitatively, the model alternates between consensus steps (K samples agree) and exploration steps (high-entropy forks), echoing findings that high-entropy tokens are key RLVR fork points (Wang et al., 2025).

Takeaway

Multiplex Thinking offers a middle path: keep discrete sampling randomness so RL can train, but let continuous vectors carry multiple candidates per step. For test-time scaling, it adds a new axis — not just sampling more rollouts, but making each step's representation richer. As the author puts it: instead of a scribe writing every thought down, the model becomes a detective who clones itself at every fork and compresses all leads into one high-density capsule — the multiplex token.

References

1. Tang, Y. et al. (2026). *Multiplex Thinking: Reasoning via Token-wise Branch-and-Merge*. arXiv:2601.08808v1. 2. Wei, J. et al. (2022). *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*. 3. Zhang et al. (2025). *Soft Thinking*. 4. Lambert et al. (2025). *Reinforcement Learning with Verifiable Rewards (RLVR)*. 5. Shao et al. (2024). *Group Relative Policy Optimization (GRPO)*.