13.1 Gaussian Mixtures
Consider two populations with means \(\mu_1\) and \(\mu_2\) and the same variance, say \(\sigma^2=1\), and let \(X\) be the observed outcome (from one of the two populations), and $ Z {0, 1}$ be a hidden variable (not observable) that indicates the population label, with \(P(Z=1)=\pi\).
Using only the observed data, \(\{x_i\}_{i=1}^{n}\), we want to estimate the two population means and the mixing probability \(\theta=(\mu_1, \mu_2, \pi)\). The density of \(X\) is a mixture of two Gaussian random variables, i.e.
\[p_X(x) = (1-\pi) \;\phi_{\mu_1}(x) + \pi \; \phi_{\mu_2}(x)\] where \(\phi_{\mu}\) is the density function of \(\mathcal{N}(\mu,1)\).
The log-likelihood function based on \(n\) observed training data computes as \[\ell(\mathbf{x}|\theta) = \sum_{i=1}^{n} \log \Bigl( (1-\pi)\phi_{\mu_1}(x_i) + \pi \phi_{\mu_2}(x_i)\Bigr)\]
We can solve this by gradient descent, but it is often slow. Instead, we can treat the labels from which distribution each sample comes from (call them \(Z\)) as missing and then we can use the EM algorithm.