Neural Thickets: When Random Guessing Makes AI Smarter — Deep Dive into MIT's RandOpt

Forum topic · 小凯 · 2026-03-19

Summary

A Chinese tech forum post analyzes MIT's paper 'Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights.' The key finding: around sufficiently pretrained large language models, randomly perturbed weights frequently yield 'task experts' that excel at specific downstream tasks. On Qwen2.5 models scaled from 0.5B to 32B parameters, the fraction of random Gaussian perturbations matching or exceeding baseline GSM8K performance rose from 0% to 64%. The authors define 'solution density' and 'spectral discordance' metrics showing both density and task-specialization follow scaling laws, and identify three regimes: needle-in-a-haystack, thicket, and plateau. Building on this, they propose RandOpt, a fully parallel algorithm that samples N Gaussian perturbations, selects the best K, and ensembles them via majority voting. With 200 GH200 GPUs on Olmo-3-7B-Instruct, RandOpt reached 70% accuracy on Countdown in 3.2 minutes, matching PPO, GRPO, and ES on math, code, writing, and chemistry benchmarks. Limitations include reliance on strong pretraining, saturating gains, and K-fold inference cost (mitigable via distillation). The post connects the findings to meta-learning (MAML), the Baldwin effect, AI safety, and federated learning.

*Translation and analysis of a zhichai.net forum post discussing MIT's paper 'Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights' (arXiv:2603.12228).*

From Needle-in-a-Haystack to Thicket

In 2001, Schmidhuber, Hochreiter, and Bengio argued that "random guessing cannot be regarded as a legitimate learning algorithm" — in a billion-dimensional parameter space, finding a good solution by chance seemed hopeless. Twenty-five years later, MIT researchers discovered that once a model is sufficiently pretrained, the picture flips: capable "task experts" cluster densely around pretrained weights like a thick thicket of bushes, not rare needles in a haystack.

Key experimental findings

Scaling law for solution density: On Qwen2.5 models scaled from 0.5B to 32B parameters, the fraction of random Gaussian perturbations matching or beating baseline performance rose from 0% to 64% on GSM8K and from 8% to 60% on Countdown.
Experts, not generalists: The researchers introduced Spectral Discordance, a metric showing that perturbations improving one task often degrade others. Discordance increases with scale — the perturbation space fills with specialists (math experts, chemistry experts, writing experts), which form distinct clusters under PCA projection.
RGB landscape visualizations confirm task-specific performance regions in 2D slices of parameter space, refuting the "generalist" hypothesis.

Three regimes (minimal MLP experiment)

1. Needle-in-a-haystack: Untrained models — random perturbations are useless; gradient-based search is required. 2. Thicket: Mixed-signal pretraining — perturbations produce diverse, useful specialists for each signal type. 3. Plateau: When test data matches pretraining data — the pretrained weights are already near-optimal, and perturbation only hurts.

This suggests pretraining does more than fill a knowledge base: it *locates* the model at a strategic point from which adaptation paths to many tasks are short — an automatic realization of MAML-like meta-learning, echoing the Baldwin effect in evolutionary biology.

RandOpt: Random Optimization

The proposed algorithm is remarkably simple:

1. Sample N Gaussian perturbations from pretrained weights 2. Evaluate each on training data (fully parallel) 3. Select the top K perturbations 4. Ensemble at test time via majority voting

No gradients, no backpropagation, no sequential updates. Wall-clock complexity is O(1) instead of O(T) for PPO, GRPO, or ES.

Headline result

Using 200 GH200 GPUs, Olmo-3-7B-Instruct, N=2000 perturbations, and K=50 experts, RandOpt achieved 70% accuracy on Countdown in 3.2 minutes — competitive with mainstream RL methods across Countdown, GSM8K, MATH-500, OlympiadBench, MBPP, ROCStories, and USPTO.

The role of ensembling

K=1 is usually insufficient; K=50 consistently outperforms. Inference cost grows K-fold, but the authors show distillation can recover most performance in a single model. On GSM8K (86.7% with K=50), gains decompose into a reasoning thicket (+12.3%, genuine reasoning improvement) and a format thicket (+19.0%, better output formatting).

Limitations

Requires pretraining: From random initialization, solution density is too low — RandOpt is a post-training method, not a replacement for pretraining.
Saturating gains: Improvements plateau with scale, N, and K; gains far beyond baseline likely require leaving the thicket (structured search again).
Inference cost: K models must run at test time unless distilled.

Implications

For research: Complex post-training algorithms (RLHF, DPO, PPO) may partly be unnecessary near strong pretrained weights; pretraining quality may matter more than post-training sophistication.
For engineering: Fully parallel, communication-free sampling suits bandwidth-constrained settings and federated learning.
For safety: Dense, diverse experts around pretrained weights imply latent capabilities (both beneficial and harmful) beyond what benchmarks reveal.
Philosophical angle: Learning may be less about searching and more about *positioning* — intelligence as standing where answers are easy to reach.

References

1. Gan, Y., et al. (2026). Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights. *arXiv preprint arXiv:2603.12228*. Project page: https://thickets.mit.edu 2. Schmidhuber, J., Hochreiter, S., & Bengio, Y. (2001). Evaluating benchmark problems by random guessing. *NeurIPS*. 3. Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. *ICML*. 4. Salimans, T., et al. (2017). Evolution strategies as a scalable alternative to reinforcement learning. *arXiv:1703.03864*. 5. Baldwin, J. M. (1896). A new factor in evolution. *The American Naturalist*, 30(354).