Collaborative Parallel Thinking (CPT): Breaking Information Silos in Parallel Reasoning

> Parallel reasoning used to be like sending ten people into the same maze, each with a blank map, all starting from scratch. CPT says: pin your map fragments to a bulletin board so later explorers can see them. No training, no model changes—just broadcast what is already known.

1. The Information Silos of Parallel Reasoning

Test-Time Scaling (TTS) allocates more compute during inference to improve complex reasoning (math, coding, science). It has two routes:

1. Sequential scaling: one long chain of thought (Long CoT, e.g., DeepSeek-R1 style reasoning). 2. Parallel scaling: multiple independent branches explored simultaneously, then voted or selected—e.g., Best-of-N, Self-Consistency, Tree-of-Thought.

Parallel scaling is naturally parallelizable: ten chains can run concurrently on GPUs, with wall-clock latency determined by the longest chain. But it has a hidden cost: information silos. Branches cannot tell each other "that path is a dead end" or "this constraint is already derived," so they massively duplicate work. The cited paper shows redundant information grows over time in independent parallel sampling—the more branches and the longer they run, the more they repeat each other.

2. CPT's Three Mechanisms

Information Extraction

Instead of sharing raw (long, noisy) reasoning traces, CPT uses the policy model π as an extractor. After each branch generates a chunk of C tokens, the model distills compact information units: intermediate conclusions ("we proved x=2"), constraints, observations, counterexamples.

Key insight: share thinking *results*, not thinking *processes*—meeting minutes, not meeting recordings.

Deduplicated Information Pool

CPT maintains a deduplicated query-level pool P. New units are compared against the pool via text embedding similarity; if the max similarity exceeds a threshold τ_dup = 0.75, the candidate is discarded. Ten branches discovering "x is even" adds no more information than one—only *unique* fragments should be kept to maximize information density.

Adaptive Broadcast Scheduling

• Probe phase: branches explore independently for a few steps while the pool accumulates; per-step new units n^t yield a windowed average gain g_j and relative gain r_j = g_j / (g_1 + ε).
• Broadcast phase: when r_j exceeds threshold, up to M = 512 pooled items are serialized into a shared context block B injected into every branch's input before the next step. When r_j falls, broadcasting stops—early silence (build foundations), mid-phase sharing (richest gains), late silence (diminishing returns).

3. Full Workflow (AIME example, K=8 branches, C=512-token chunks)

1. Init: all branches see the same problem; pool and shared block empty. 2. Probe (steps 1–3): independent generation, extraction, deduplication, gain tracking. 3. Broadcast (steps 4–7): sample up to 512 items into B, inject into all contexts; branches build on shared conclusions; pool keeps updating. 4. Stop or continue: broadcasting halts when gains drop; early termination if a branch solves it. 5. Vote/verify: final answers collected and voted on, as in standard parallel methods.

4. Experimental Results

Tested on HMMT and AIME (24/25/26) with Qwen3-4B-Thinking and Qwen3-30B-A3B-Thinking, against Base Parallel Sampling, DeepConf, and LeaP:

• CPT establishes a better accuracy–latency Pareto frontier on all models and datasets, consistently across rollout budgets and model sizes.
• Redundant discoveries per step drop significantly while new-information discovery stays comparable.
• CPT also dominates on accuracy-vs-token-count Pareto frontiers—shared facts don't need to be re-derived ten times.
• Information extraction and dedup add only moderate overhead; however, because broadcasting requires re-prefilling updated contexts, CPT is not always best under a pure FLOPs budget.
• Information-theoretic view: effective width of K independent branches is K_eff = K / (1 + (K−1)ρ), where ρ is cross-path correlation. High ρ (similar reasoning) yields diminishing returns; CPT lowers ρ by sharing, raising effective width.

5. Limitations and Future Directions

Limitations:

• Synchronous steps: information found at step t is only usable at step t+1.
• Re-prefilling after each broadcast incurs FLOPs overhead.
• Hyperparameter sensitivity (M = 512, τ = 0.75 were tuned per setting; no auto-tuning).

Future directions: 1. Cache-aware sharing: inject pool embeddings as extra KV cache instead of re-prefilling text. 2. Attention-level sharing: branches attend directly to a shared "information-pool" KV head. 3. Asynchronous broadcast: real-time pool updates to reduce discovery-to-use latency.

6. Why CPT Matters

The core insight: parallel reasoning is not ten independent individuals but a team. Without sharing, branches converge on similar paths (high ρ)—ten branches may equal only 3–4 effective ones. CPT shows diversity comes from sharing, not isolation: shared information exists to avoid repetition and free resources for unexplored search space, not to force consensus. Crucially, CPT is fully training-free—any existing reasoning model can adopt it at inference time. As TTS becomes the dominant reasoning paradigm, CPT points toward compute that not only runs more chains but makes the chains talk to each other.

References

1. Wang, X., et al. (2026). Share More, Search Less: Collaborative Parallel Thinking for Efficient Test-Time Scaling. *arXiv:2605.27030*. 2. Snell, J., et al. (2024). Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters. *NeurIPS*. 3. Fu, Y., et al. (2025). DeepConf: Confidence-based Pruning for Parallel Test-Time Scaling. *ICML*. 4. Liu, B., et al. (2025). LeaP: Learning from Peers in Parallel Reasoning. *OpenReview*. 5. Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. *NeurIPS*.