Avoiding Singularities and Spurious Maximizers in the Maximum Likelihood Estimation of Gaussian Mixture Models: Relative Class Constraints Enabled by Metaheuristic Optimization

Abstract

This research addresses a long-standing challenge in the maximum likelihood estimation of finite mixture models: the emergence of singularities and spurious local maxima due to an unbounded likelihood function. Traditional optimization methods, such as the Expectation-Maximization (EM) algorithm, are highly sensitive to initial values and often converge to degenerate solutions. While constraints on the parameter space are essential to prevent convergence to these degeneracies, the EM algorithm cannot accommodate what the literature suggests are the most effective type, relative constraints across classes. To address this shortcoming and overcome these limitations, Simulated Annealing with Predicted Constraints (SAPC) is introduced as an optimization framework that incorporates relative class constraints derived from class separation metrics, including the Jensen-Shannon (JS) distance. By integrating these constraints, SAPC confines the search to a compact region of the parameter space that is more likely to contain the true model parameters. These constraints, which are predicted directly from the input data, also extend naturally to tests of class enumeration. Specifically, if the predicted class separation intervals for any pair of classes contain the 95% cutoff expected for data generated from a single class (empirically determined to be 0.084 for 2D data), the hypothesis that these two classes represent a single class cannot be rejected and, consequently, the hypothesis that the data contain G-1 classes cannot be rejected. Simulation studies compare the SAPC method to the EM algorithm, evaluating class enumeration accuracy and parameter bias. Results indicate that SAPC significantly reduces parameter bias in the low class separation conditions.