QUANTIZATION AND APPROXIMATION OF NEURAL NETWORKS

Abstract

This thesis investigates several fundamental mathematical problems arising in machine learning, with a specific focus on the theoretical properties of neural networks. The first part of this work addresses neural network quantization. Despite the widespread use of quantization as a primary method for compressing deep neural networks, a majority of existing approaches lack provable guarantees. By leveraging tools from frame theory and signal processing, we establish a neural network quantization framework that provides rigorous error estimates. Furthermore, we explore the universal approximation theory of quantized networks, focusing on the most extreme case: one-bit neural networks. The second focus of this thesis pertains to the theoretical analysis of neural network optimization, supported by the foundations of neural network approximation theory. We investigate the optimization of shallow neural networks to determine how the regularity of the target function influences the convergence rate. Finally, we analyze the neural collapse phenomenon under the unconstrained feature model setting, an analytical framework justified by approximation theory.