Paper Overview
Field: Machine Learning Author: Shijun Zhang Published: 2025-09-01 arXiv: 2509.00142
Abstract
Many parameter-efficient methods generate the parameters of a large neural network from a low-dimensional latent representation. Given an architecture \(\Phi\) with \(P_\Phi\) parameter slots, we write \(\boldsymbol\theta_f=\mathcal{G}(\boldsymbol\xi_f)\), where \(\mathcal{G}\colon\mathbb{R}^M\to\mathbb{R}^{P_\Phi}\) is a parameter generator and \(\boldsymbol\xi_f\in\mathbb{R}^M\) is a latent representation of the target function \(f\). The architecture \(\Phi\) and the generator \(\mathcal{G}\) are shared across the entire target class, while each target \(f\) is represented by its own latent vector \(\boldsymbol\xi_f\), with \(\Phi_{\mathcal{G}(\boldsymbol\xi_f)}\) approximating \(f\). This framework encompasses hypernetworks, low-dimensional parameterizations, parameter-efficient adaptation, and model compression. Understanding the trade-off between the latent dimension \(M\) and the network budget \(P\) is therefore essential for characterizing the expressive efficiency of these methods.
We study this trade-off for affine generators and fully connected ReLU architectures. More precisely, jointly optimizing over architectures \(\Phi\) satisfying \(P_\Phi \le P\) and affine generators \(\mathcal{G}\colon\mathbb{R}^M\to\mathbb{R}^{P_\Phi}\), we prove that the optimal worst-case uniform approximation error over the unit ball of \(\alpha\)-Holder functions on \([0,1]^d\) has the exact order \((P\cdot\min\{M,P\})^{-\alpha/d}\), where \(0 < \alpha \le 1\).
In particular, our results show that even a fixed-dimensional latent space suffices to achieve vanishing approximation error as the network budget increases.
---
*Auto-collected on 2026-09-02*