In machine learning and statistics, EM Algorithm is the abbreviation for Expectation Maximization Algorithm, which is used to handle latent variables. GMM Model means Gaussian mixture model. In this article, we'll have an introduction on how to use EM Algorithm to handle GMM model.

Background

edit

First let's warm up with a simple scenario. In the picture below, we have the Red Blood Cell Hemoglobin Concentration and the Red Blood Cell Volume data of two groups of people, the Anemia Group and the Control Group(i.e. the group of people without Anemia). It's clear that people with Anemia have lower red blood cell volume and lower red blood cell hemoglobin concentration than those without Anemia.

 
GMM model with labels

To make it simple, let   be a random vector:   And denote   as the group where   belongs.(  when   belongs to Anemia Group and   when   belongs to Control Group). And from medical knowledge, we believe that   are normally distributed in each group, i.e.  . Also  , where  (in this scenario,  ). Now we'd like to estimate  .

We can use maximum likelihood estimation on this question. The log likelihood function is shown below.

 

As we know the   for each  , the log likelihood function can be simplified as below:

 

Now we can maximize the likelihood function by making partial derivative over  . Since this step only involves some simple algebra calculation, I'll directly show the result.

 

 

  [1]

In the example above, we can see that if   is known to us, the estimation of parameters can be quite simple with maximum likelihood estimation. But what if   is unknown to us? It'll be hard to estimate the parameters. [2]

 
GMM without labels


In this case, we call   a latent variable(i.e. not observed). With unlabled scenario, we need the Expectation Maximization Algorithm to estimate $z$ as well as other parameters. Generally, we would name the problem setting above as Gaussian Mixture Models(i.e. GMM) since the data in each group is normally distributed. [3]

In a general circumstance in machine learning, we can see the latent variable   as some latent pattern lying under the data, which we cannot see very directly. And we can see   as our data,   as the parameter of the model. With EM algorithm, we may find some underlying pattern   in the data  , along with the estimation of parameters. The wide application of this circumstance in machine learning makes EM algorithm very important. [4]

EM Algorithm in GMM

edit

The Expectation Maximization Algorithm consists of two steps: the E-step and the M-step. Firstly, we can randomly initialize the value of our model parameters and the   In the E-step, the algorithm tries to guess the value of   based on the parameters. In the M-step, the algorithm updates the value of the model parameters based on the guess of   in the E-step. These two steps will repeat until convergence. Let's see the algorithm in GMM first.

Repeat until convergence: {

   1. (E-step) For each  , set 
    
   2. (M-step) Update the parameters
    
       
       

} [1]


We can take a closer look at the E-step. In fact, with Bayes Rule, we can get the following result:  

According to GMM setting, we can have these following formulas:    

In this way, we can clearly move on to the E-step and M-step according to the randomly initialized parameters.


References

edit
  1. ^ a b Ng, Andrew. "CS229 Lecture notes" (PDF).
  2. ^ Hui, Jonathan (13 October 2019). "Machine Learning —Expectation-Maximization Algorithm (EM)". Medium.
  3. ^ Tong, Y. L. (2 July 2020). "Multivariate normal distribution". Wikipedia.
  4. ^ Misra, Rishabh (7 June 2020). "Inference using EM algorithm". Medium.

Category:Machine learning