Estimating Expected Outputs of Wide Random MLPs More Efficiently Than Monte Carlo Sampling

Forum topic · 小凯 · 2026-05-08

Summary

A new paper by Wilson Wu, Victor Lecomte, Michael Winer, George Robinson, Jacob Hilton, and Paul Christiano (arXiv:2605.05179) presents a method for estimating the expected output of an MLP at initialization over Gaussian inputs without running any samples through the network. Instead of empirical averaging over sampled inputs, the approach builds approximate representations of activation distributions at each layer using cumulants and Hermite expansions. The authors show theoretically and empirically that for sufficiently wide networks, their estimator reaches a target mean squared error with substantially fewer FLOPs than Monte Carlo sampling. The method is especially effective at estimating probabilities of rare events and can also be applied to model training. The findings suggest a production path for models with greatly reduced probability of catastrophic tail risks.

Paper Overview

Field: Machine Learning Authors: Wilson Wu, Victor Lecomte, Michael Winer, George Robinson, Jacob Hilton, Paul Christiano Published: 2026-05-06 arXiv: 2605.05179

Abstract

By far the most common way to estimate an expected loss in machine learning is to draw samples, compute the loss on each one, and take the empirical average. However, sampling is not necessarily optimal. Given an MLP at initialization, the authors show how to estimate its expected output over Gaussian inputs without running samples through the network at all. Instead, they produce approximate representations of the distributions of activations at each layer, leveraging tools such as cumulants and Hermite expansions.

Key Findings

The proposed estimator achieves a target mean squared error using substantially fewer FLOPs than Monte Carlo sampling, for sufficiently wide networks, with both theoretical and empirical support.
The method performs particularly well at estimating the probabilities of rare events.
The techniques can additionally be used for model training.
Together, these findings suggest a path to producing models with a greatly reduced probability of catastrophic tail risks.

Discussion

This work challenges the default assumption that Monte Carlo sampling is the right tool for estimating expected losses or outputs. By propagating approximate activation distributions layer by layer (via cumulants and Hermite expansions), the method replaces forward passes with cheaper analytical computations. The strong performance on rare-event estimation is especially notable for AI safety, since catastrophic tail risks are by definition hard to measure via sampling.