NEW EFFICIENT ALGORITHMS FOR NESTED MACHINE LEARNING PROBLEMS

Abstract

In recent years, machine learning (ML) has achieved remarkable success by training large-scale models on vast datasets. However, building these models involves multiple interdependent tasks-such as data selection, hyperparameter tuning, and model architecture search-that can lead to nested objectives when optimized jointly. These nested objectives arise because each task both influences and depends on the others. This dissertation aims to develop efficient algorithms to tackle these challenging nested problems in machine learning.

In the first part, we formalize nested ML problems as bilevel optimization tasks and presenting efficient algorithms with theoretical guarantees that solve them. Then, in the second part, we extend to the federated/distributed learning context, examining how algorithmic designs must be adapted to meet the challenges of that environment. Finally, in the third part, we cover challenges with hierarchies in the distributed learning setting including data cleaning, network pruning and constrained problems.