Paper Overview
Field: Machine Learning Author: Shijun Zhang Published: 2025-09-01 arXiv: 2509.00142
Summary
Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture Φ with P_Φ parameter slots, we write θ_f = G(ξ_f), where G: R^M → R^{P_Φ} is a parameter generator and ξ_f ∈ R^M is a latent representation of the target function f. The architecture Φ and the generator G are shared across the entire target class, while each target f is represented by its own latent vector ξ_f, with Φ_{G(ξ_f)} approximating f. This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression.
Understanding the trade-off between latent dimension M and network budget P is therefore essential for characterizing the expressive efficiency of these methods. The paper studies this trade-off for affine generators and fully connected ReLU architectures. More precisely, jointly optimizing over architectures Φ satisfying P_Φ ≤ P and affine generators G: R^M → R^{P_Φ}, it is proved that the optimal worst-case uniform approximation error on the unit ball of α-Hölder functions on [0,1]^d has exact order (P · min{M, P})^{-α/d}, where 0 < α ≤ 1.
In particular, the results show that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget P increases.
Original Abstract (excerpt)
> Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture Φ with P_Φ parameter slots, we write θ_f = G(ξ_f), where G: R^M → R^{P_Φ} is a parameter generator and ξ_f ∈ R^M is a latent representation of the target function f. ...
--- *Auto-collected on 2026-09-02*