EM algorithm and GMM model

(Redirected from EM Algorithm And GMM Model)

In statistics, EM (expectation maximization) algorithm handles latent variables, while GMM is the Gaussian mixture model.

Background

edit

In the picture below, are shown the red blood cell hemoglobin concentration and the red blood cell volume data of two groups of people, the Anemia group and the Control Group (i.e. the group of people without Anemia). As expected, people with Anemia have lower red blood cell volume and lower red blood cell hemoglobin concentration than those without Anemia.

 
GMM model with labels

  is a random vector such as  , and from medical studies[citation needed] it is known that   are normally distributed in each group, i.e.  .

  is denoted as the group where   belongs, with   when   belongs to Anemia Group and   when   belongs to Control Group. Also   where  ,   and  . See Categorical distribution.

The following procedure can be used to estimate  .

A maximum likelihood estimation can be applied:

 

As the   for each   are known, the log likelihood function can be simplified as below:

 

Now the likelihood function can be maximized by making partial derivative over  , obtaining:

 
 
 [1]

If   is known, the estimation of the parameters results to be quite simple with maximum likelihood estimation. But if   is unknown it is much more complicated.[2]

 
GMM without labels

Being   a latent variable (i.e. not observed), with unlabeled scenario, the Expectation Maximization Algorithm is needed to estimate   as well as other parameters. Generally, this problem is set as a GMM since the data in each group is normally distributed. [3][circular reference]

In machine learning, the latent variable   is considered as a latent pattern lying under the data, which the observer is not able to see very directly.   is the known data, while   are the parameter of the model. With the EM algorithm, some underlying pattern   in the data   can be found, along with the estimation of the parameters. The wide application of this circumstance in machine learning is what makes EM algorithm so important.

EM algorithm in GMM

edit

The EM algorithm consists of two steps: the E-step and the M-step. Firstly, the model parameters and the   can be randomly initialized. In the E-step, the algorithm tries to guess the value of   based on the parameters, while in the M-step, the algorithm updates the value of the model parameters based on the guess of  of the E-step. These two steps are repeated until convergence is reached.

The algorithm in GMM is:

Repeat until convergence:

   1. (E-step) For each  , set
    
   2. (M-step) Update the parameters
    
       
       

[1]

With Bayes Rule, the following result is obtained by the E-step:

 

According to GMM setting, these following formulas are obtained:

 

 

In this way, a switch between the E-step and the M-step is possible, according to the randomly initialized parameters.

References

edit
  1. ^ a b Ng, Andrew. "CS229 Lecture notes" (PDF).
  2. ^ Hui, Jonathan (13 October 2019). "Machine Learning —Expectation-Maximization Algorithm (EM)". Medium.
  3. ^ Tong, Y. L. (2 July 2020). "Multivariate normal distribution". Wikipedia.