Rare Event Estimation via Iterative Unalignment: Importance Sampling Through Weight Perturbation

Forum topic · 小凯 · 2026-09-23

Summary

A new arXiv paper (2609.24969) by Hanming Yang, Daksh Mittal, Jing Dong, and Hongseok Namkoong addresses the problem of estimating the probability of rare events caused by stochastic variation in LLM agent behavior. Since naive Monte Carlo sampling is computationally infeasible over the combinatorially vast space of agent trajectories, the authors propose a novel importance sampling (IS) method that constructs the proposal distribution by perturbing the original model's weights. The proposal itself is a differentiable, parameterized language model, enabling gradient-based search in weight space. Their objective combines a differentiable surrogate for event amplification with an adaptive regularization scheme that balances amplification against estimator stability. The method was evaluated on models of roughly 120 million and 2.6 billion parameters across three event families and more than 300 rare events with probabilities as low as 10^-9, with reference probabilities estimated at under 10% relative standard error. In the most verifiable settings, the IS estimator achieved over 800x computational efficiency gains over naive Monte Carlo for events with probability below 10^-7. Code is open-sourced at the namkoong-lab GitHub repository.

Paper Overview

Field: Machine Learning Authors: Hanming Yang, Daksh Mittal, Jing Dong, Hongseok Namkoong Published: 2026-09-21 arXiv: 2609.24969

Abstract

As agents are deployed with increased autonomy, even extremely rare events along their stochastic output trajectories can occur and prove catastrophic. Safe deployment therefore does not depend on whether these events can occur, but on how often they might. The paper studies the problem of estimating the probability of rare events that arise from stochastic variation in the agent's own actions.

Estimating this type of risk requires searching over the combinatorially vast space of trajectories. Naive Monte Carlo is computationally prohibitive in this regime, and constructing effective importance sampling (IS) proposals requires coordinated changes to a context-dependent chain of conditional distributions.

Method

The authors develop a new IS method that perturbs the original model's weights to construct the proposal distribution. Key aspects:

The proposal distribution is itself a differentiable, parameterized language model, supporting gradient-based search in weight space.
The objective function combines a differentiable surrogate for event amplification with an adaptive regularization scheme that dynamically balances amplification against estimator stability.

Results

Evaluated on models of approximately 120 million and 2.6 billion parameters.
Over 300 rare events across three event families, with probabilities as low as 10^-9; reference probabilities computed with relative standard error below 10%.
In the most verifiable scenario, the IS estimator achieved more than 800x computational efficiency improvement over naive Monte Carlo for events with probability below 10^-7.

Code

Open source: https://github.com/namkoong-lab/iterative-unalignment

---

*Originally compiled on 2026-09-23*