Neural Thickets: When Random Guessing Makes AI Smarter — Deep Dive into MIT's RandOpt

*Translated and structured summary of a long-form Chinese forum post analyzing MIT's 2026 paper "Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights" (arXiv:2603.12228).*

Key points

• From needle-in-a-haystack to thicket: In 2001, Schmidhuber, Hochreiter, and Bengio argued that "random guessing cannot be considered a reasonable learning algorithm." MIT's 2026 paper shows this is only half true: around *sufficiently pretrained* large language models, good solutions are densely clustered near the pretrained weights — a "neural thicket" of task experts.
• Solution density scales: The authors define *solution density* — the fraction of random Gaussian perturbations of pretrained weights that improve performance on a task. On GSM8K, this fraction rose from 0% at 0.5B parameters to 64% at 32B (Qwen2.5 series); on Countdown, from 8% to 60%. Density grows with model scale, pretraining data volume, and pretraining quality — a scaling law.
• Experts, not generalists: A newly introduced metric, *Spectral Discordance*, shows sampled solutions are task-specific specialists (near 1) rather than uniform generalists (near 0), and discordance increases with scale. PCA projections of per-task performance vectors reveal distinct clusters — math experts, chemistry experts, writing experts.
• RandOpt algorithm: Sample N Gaussian perturbations, evaluate all in parallel, keep the top K, and ensemble them via majority voting at test time. No gradients, no backpropagation. Wall-clock training complexity is O(1) versus O(T) for sequential methods like PPO, GRPO, or evolution strategies.
• Speed: On a 200-GPU GH200 cluster, using Olmo-3-7B-Instruct with N=2000 samples and K=50, RandOpt reached 70% accuracy on Countdown in 3.2 minutes. Across benchmarks (Countdown, GSM8K, MATH-500, OlympiadBench, MBPP, ROCStories, USPTO) it matched or sometimes beat mainstream RL methods.
• Ensembling matters: K=50 clearly outperforms K=1, leveraging "wisdom of the crowd" across diverse experts. Inference costs K× more, but distillation from the ensemble back into a single model can recover most performance at normal inference cost.
• Why thickets form: A minimal MLP experiment on 1D signal prediction shows three regimes: (1) *needle-in-a-haystack* for un-pretrained models, (2) *thicket* for models pretrained on mixed signal types, (3) *plateau* when the test task matches pretraining exactly. Pretraining appears to *position* the model at a strategic point in parameter space — an emergent meta-learning effect reminiscent of MAML and the Baldwin effect.
• Reasoning vs. format thickets: On GSM8K, RandOpt (K=50) reached 86.7%; gains decompose into 12.3% genuine reasoning improvements and 19.0% format corrections — the thicket contains both reasoning experts and formatting experts.

Limitations

• RandOpt only works on well-pretrained models; from random initialization, solution density is far too low.
• Performance gains saturate with N, K, and model size; escaping beyond the thicket likely requires gradient-based search again.
• Inference cost scales with ensemble size; distillation mitigates but does not eliminate this.
• The method is a *complement* to gradient descent — pretraining itself still requires traditional optimization to reach the thicket in the first place.

Implications

• Post-training may be simpler than assumed: once pretrained representations exist, even random search rivals carefully engineered RL pipelines.
• Fully parallel, low-communication training: RandOpt suits bandwidth-constrained distributed settings, including federated learning where gradient synchronization is impractical.
• AI safety angle: dense expert neighborhoods imply pretrained models harbor more diverse latent capabilities (both beneficial and harmful) than benchmarks reveal.
• Broader connections: fitness landscapes in evolutionary biology, possible phase-transition-like thresholds in scale, and extensions to image generation (color thickets per Appendix J).

References cited in the original post

1. Gan, Y., et al. (2026). *Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights*. arXiv:2603.12228. 2. Schmidhuber, J., Hochreiter, S., & Bengio, Y. (2001). Evaluating benchmark problems by random guessing. *NeurIPS*. 3. Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. *ICML*. 4. Salimans, T., et al. (2017). Evolution strategies as a scalable alternative to reinforcement learning. arXiv:1703.03864. 5. Baldwin, J. M. (1896). A new factor in evolution. *The American Naturalist*, 30(354), 441-451.

Project page: https://thickets.mit.edu