Procedural Adversarial Learning: Finding Neural Network Failures with Procedural Generators

Forum topic · 小凯 · 2026-11-05

Summary

Researchers Jack Nugent and Jia Deng introduce Procedural Adversarial Learning (PAL), a minimax training and adversarial evaluation framework that searches the input space of procedural generators for difficult instances where neural networks fail. A procedural generator is an interpretable program that maps a low-dimensional vector of control parameters (layout, lighting, pose) to a rendered image and its label. The authors present Simple-PAT, an algorithm that uses zeroth-order optimization in the generator's parameter space to produce adversarial training samples on demand. They also build PAL-Dojo, a collection of efficient procedural generators for surface normal estimation, image segmentation, and human pose estimation. Against black-box adversaries performing 50 million aggregate queries across models, Simple-PAT reduces worst-case error by a geometric mean of 3.15x compared to standard training and 2.63x compared to the strongest robust training baseline. Paper: arXiv 2611.06890; code available on GitHub (princeton-vl).

Paper Overview

Field: Computer Vision (CV)
Authors: Jack Nugent, Jia Deng
Published: 2026-11-04
arXiv: 2611.06890

Abstract

For many computer vision tasks, neural networks can be trained to produce a low average error, yet they still fail on seemingly simple scenes. Reducing these failures requires first finding them, but they may not be in a fixed dataset. The authors study Procedural Adversarial Learning (PAL), a framework of minimax training and adversarial evaluation which searches the input space of a procedural generator for difficult instances.

The procedural generator is an interpretable program mapping a low-dimensional vector of control parameters (e.g. layout, lighting, pose) to a rendered image and its label.

Key Contributions

Simple-PAT algorithm: uses zeroth-order optimization in the procedural generator's parameter space to propose adversarial training samples that are generated on demand.
PAL-Dojo: a collection of efficient procedural generators targeting surface normal estimation, image segmentation, and human pose estimation.

Results

When evaluated against black-box adversaries performing 50 million aggregate queries across models, Simple-PAT reduces worst-case error by:

3.15x (geometric mean) compared to standard training
2.63x compared to the strongest robust training baseline

Resources

Paper: arXiv 2611.06890
Code: https://github.com/princeton-vl/procedural-adversarial-learning

--- *Auto-collected on 2026-11-06*