Advances in Variational Bayes Theory: Adaptation, Uncertainty Quantification, and Amortization

Abstract

Variational methods and variational inference have been widely used in statistical physics, Bayesian posterior approximation, and modern generative modeling. At a high level, a variational method transforms an often difficult or intractable inference problem into an optimization problem by projecting a target distribution, under an appropriate choice of divergence, onto a simpler and more tractable family known as the variational family. This approximation involves three main ingredients: the choice of the variational family, which affects both approximation accuracy and computational cost; the choice of divergence; and the optimization procedure itself, which is often nonconvex and does not guarantee global optimality. Consequently, efficient and scalable algorithms are highly desirable.

Since variational inference is ultimately an approximation to the target distribution, some loss is unavoidable. This naturally raises fundamental statistical questions about how to quantify such loss. In the Bayesian literature, one important line of work studies the frequentist behavior of the variational posterior, including its posterior contraction properties and uncertainty quantification. Because variational inference typically relies on a restricted variational family, such as the popular mean-field class, valid uncertainty quantification is generally not guaranteed. One chapter of this thesis proposes a general variational bagging framework to achieve valid uncertainty quantification for a broad class of statistical models under a variational Bayes framework.

A related challenge concerns adaptation to the unknown complexity of the true data-generating mechanism. From this perspective, Bayesian inference is especially appealing because it can adapt across a collection of candidate models through hierarchical prior design: one places a prior over models and then a conditional prior on the parameters within each selected model, allowing posterior inference to automatically adapt to model complexity. However, both Bayesian adaptation and variational Bayes adaptation typically require searching over a broad range of model complexities, including unnecessarily large models even when the true data-generating process is relatively simple. In the absence of prior knowledge, this can lead to substantial computational inefficiency. Chapter 2 of this thesis proposes an early-stopping mechanism for variational Bayes adaptation. This framework is further extended to variational empirical Bayes and to frequentist penalized estimators.

At the same time, modern applications demand not only statistical validity but also computational scalability. This raises the question of whether the cost of inference can be amortized across observations or tasks, rather than repeatedly solving a full optimization problem from scratch. This is precisely the idea of amortized inference: one incurs a larger upfront cost to learn a reusable inference or optimization mechanism, so that future inference tasks can be carried out much more efficiently. Chapter 5 of this thesis focuses on amortized variational Bayes.

In summary, this dissertation advances the theory and practice of variational Bayes along three broad directions: adaptation, uncertainty quantification, and amortization. It studies how variational procedures can be made more reliable for uncertainty quantification, more computationally efficient for adaptive inference, and more scalable through amortization. Together, these contributions deepen the understanding of the tradeoff between statistical accuracy and computational efficiency, and provide new tools for designing variational procedures that are both principled and practical in complex modern applications.