Rethinking Neural Thickets Without Weight Perturbations: Perturb-Select-Vote (PSV)

Overview

• Field: Computer Vision / LLMs
• Authors: Nicolas Dufour, Elouan Gardès, Amil Dravid
• Date: 2025-10-28
• arXiv: 2510.25918

Summary

The Neural Thickets hypothesis posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space, which random Gaussian perturbations can uncover. This paper reproduces the reported gains, but shows that they do not require this geometric explanation.

Key findings

• Perturbing a single decoder block already recovers much of the improvement.
• Keeping the model entirely frozen and adding a random bias to its output logits matches or exceeds full-weight perturbations on most reasoning tasks.
• This suggests an alternative explanation based on output diversity, validation-based selection, and voting.

Perturb-Select-Vote (PSV)

The authors formalize this view as Perturb-Select-Vote (PSV), a general procedure that:

1. Generates perturbed predictors; 2. Selects high-performing members on a small validation set; 3. Aggregates their predictions via voting.

This interpretation links Neural Thickets to established inference-time methods, including sampling, self-consistency, and verifier-guided selection. A first-order analysis further shows that small weight perturbations act mainly through the logit shifts they induce.

Vision experiments

The analysis extends to the vision domain: under domain shift (including Earth observation), logit PSV significantly improves performance, while full-weight perturbations provide little additional benefit.

Conclusion

Gains previously attributed to nearby weight-space experts can be obtained through output diversity, selection, and voting. Logit PSV achieves these benefits with a fully frozen model, offering a simple, training-free alternative to full-weight perturbation ensembles for both language and vision tasks.

Original Abstract (excerpt)

> The Neural Thickets hypothesis posits that pretrained language models lie in dense neighborhoods of task-specific experts in weight space, which random Gaussian perturbations can uncover. We reproduce the reported gains, but show that they do not require this geometric explanation. Perturbing a single decoder block already recovers much of the improvement, while keeping the model entirely frozen and adding a random bias to its output logits matches or exceeds full-weight perturbations on most reasoning tasks. This suggests an alternative explanation based on output diversity, validation-based selection, and voting. We formalize this view as Perturb-Select-Vote (PSV), a general procedure that generates perturbed predictors, selects high-performing members on a small validation set, and aggr...

Full paper: https://arxiv.org/abs/2510.25918