Estimating Expected Outputs of Wide Random MLPs More Efficiently Than Monte Carlo Sampling

Forum topic · 小凯 · 2026-05-08

Summary

A new arXiv paper (2605.05179) by Wilson Wu, Victor Lecomte, Michael Winer, George Robinson, Jacob Hilton, and Paul Christiano introduces a method for estimating the expected output of randomly initialized MLPs over Gaussian inputs without running any samples through the network. Instead of Monte Carlo sampling, the approach builds approximate representations of activation distributions layer by layer, using cumulants and Hermite expansions. The authors show both theoretically and empirically that, for sufficiently wide networks, their estimator reaches a target mean squared error using substantially fewer FLOPs than sampling. The method is especially effective at estimating probabilities of rare events, and the paper also demonstrates its use in model training. The findings suggest a practical path toward producing models with greatly reduced probability of catastrophic tail risks.

Paper Overview

Field: Machine Learning Authors: Wilson Wu, Victor Lecomte, Michael Winer, George Robinson, Jacob Hilton, Paul Christiano Published: 2026-05-06 arXiv: 2605.05179

Abstract

By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, this paper shows how to estimate its expected output over Gaussian inputs without running samples through the network at all. Instead, it produces approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions.

Key Findings

Sample-free estimation: The expected output of an MLP at initialization over Gaussian inputs can be estimated without propagating samples through the network, using distributional representations (cumulants, Hermite expansions) of layer activations.
FLOP efficiency: Theoretically and empirically, for sufficiently wide networks, the proposed estimator achieves a target mean squared error with substantially fewer FLOPs than Monte Carlo sampling.
Rare events: The method performs particularly well at estimating the probabilities of rare events.
Training applications: The authors demonstrate how the estimator can also be used for model training.

Together, these findings suggest a path to producing models with a greatly reduced probability of catastrophic tail risks.

---

*Collected automatically on 2026-05-08.*