English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

On the (non-)closedness of the Set of ReLU Networks

Forum topic · 小凯 · 2026-11-09

Summary

This arXiv paper (2611.09887) by Eloi Martinet and Leon Bungert studies the closedness properties of sets of ReLU neural networks in function spaces. The authors prove that the set of ReLU networks with one hidden layer and at most N neurons is weakly closed in the Sobolev space W^{1,p}(Ω) for any bounded connected open set Ω in R^d. As a consequence, every coercive, weakly lower semicontinuous energy attains a minimum over this network class, covering the Deep Ritz method with convex integrands, Sobolev training, and Lipschitz learning. They further show that as the weight decay parameter tends to zero, minimizers of the regularized problem converge up to subsequences to minimizers with minimal parameter norm. In contrast, the set of ReLU networks with two hidden layers of bounded widths is proven to be non-closed in both W^{1,p}(Ω) and C(Ω̄), thereby disproving a conjecture by Petersen, Raslan, and Voigtlaender.

On the (non-)closedness of the Set of ReLU Networks

Field: Machine Learning Authors: Eloi Martinet, Leon Bungert Published: 2026-11-06 arXiv: 2611.09887

Abstract

We prove that the set \(\mathcal{N}_N\) of ReLU networks with one hidden layer and at most \(N\in\mathbb N\) neurons is weakly closed in \(W^{1,p}(Ω)\), for a bounded connected open set \(Ω\subset\mathbb R^d\). Consequently, every coercive, weakly lower semicontinuous energy on this space attains its minimum over \(\mathcal{N}_N\). This covers the Deep Ritz method with convex integrands, Sobolev training and Lipschitz learning. Moreover, as the weight decay parameter tends to zero, minimizers of the regularized problem converge, up to subsequences, to minimizers with minimal parameter norm. We also prove that the set of ReLU networks with two hidden layers of bounded widths is not closed in \(W^{1,p}(Ω)\) or in \(C(\overlineΩ)\). This disproves a conjecture of Petersen, Raslan and Voigtlaender.

Key Contributions

  • Positive result: Single-hidden-layer ReLU networks with at most \(N\) neurons form a weakly closed set in \(W^{1,p}(Ω)\).
  • Existence of minimizers: Every coercive, weakly lower semicontinuous energy attains its minimum over \(\mathcal{N}_N\) — applicable to Deep Ritz with convex integrands, Sobolev training, and Lipschitz learning.
  • Weight decay limit: As the weight decay parameter tends to zero, regularized minimizers converge (up to subsequences) to minimizers with minimal parameter norm.
  • Negative result: ReLU networks with two hidden layers of bounded widths are not closed in \(W^{1,p}(Ω)\) or \(C(\overlineΩ)\), disproving a conjecture of Petersen, Raslan and Voigtlaender.
--- *Source: arXiv:2611.09887*

Tags

#machine-learning#relu-networks#deep-ritz-method#functional-analysis#sobolev-spaces#existence-of-minimizers#arxiv-paper

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636977