User:Jhun0324/sandbox

In Bayesian statistics, a maximum a posteriori probability (MAP) estimation is a method of estimating the parameters of a statistical model given observations by finding the parameters that equals to the mode of the posterior probability distribution. It is closely related to the method of maximum likelihood estimation (MLE), but employs an augmented optimization objective which incorporates a prior distribution (that quantifies the additional information available through prior knowledge of the distribution of a parameter) over the parameter one wants to estimate. MAP estimation can therefore be seen as a regularization of MLE.

Description

Assume we want to estimate an unobserved population parameter $\theta$ on the basis of observations, $x_{1},x_{2},...,x_{n}$. Let $f$ be the sampling distribution of $x_{i}$'s for $i=1,2,...,n$, so that $f({\displaystyle x_{1},x_{2},...,x_{n}}\mid \theta )$ is the joint density function of all observations when the underlying population parameter is $\theta$. Then the estimate:

${\hat {\theta }}_{\mathrm {MLE} }={\underset {\theta }{\operatorname {arg\,max} }}\ f(x_{1},x_{2},...,x_{n}\mid \theta )\!$

is the maximum likelihood estimate of $\theta$.

Now assume that a prior distribution $g$ over $\theta$ exists. This allows us to treat $\theta$ as a random variable as in Bayesian statistics. We can calculate the posterior probability distribution of $\theta$ using Bayes' theorem:

$f(\theta \mid x_{1},x_{2},...,x_{n})={\frac {f(x_{1},x_{2},...,x_{n}\mid \theta )\,g(\theta )}{\displaystyle \int _{\vartheta \in \Theta }f(x_{1},x_{2},...,x_{n}\mid \vartheta )\,g(\vartheta )\,d\vartheta }}\!$

where $g$ is a density function of $\theta$, $\Theta$ is the domain of $g$.

The method of MAP then estimates $\theta$ as the mode of the posterior probability distribution of this random variable:

${\hat {\theta }}_{\mathrm {MAP} }={\underset {\theta }{\operatorname {arg\,max} }}\ f(\theta \mid x_{1},x_{2},...,x_{n})={\underset {\theta }{\operatorname {arg\,max} }}\ {\frac {f(x_{1},x_{2},...,x_{n}\mid \theta )\,g(\theta )}{\displaystyle \int _{\vartheta }f(x_{1},x_{2},...,x_{n}\mid \vartheta )\,g(\vartheta )\,d\vartheta }}={\underset {\theta }{\operatorname {arg\,max} }}\ f(x_{1},x_{2},...,x_{n}\mid \theta )\,g(\theta ).\!$

The denominator of the posterior distribution (so-called marginal likelihood) is always positive and does not depend on $\theta$. Therefore, it plays no role in the optimization. Observe that the MAP estimate of $\theta$ coincides with the MLE estimate when the prior $g$ is uniform (that is, a constant function).

When the loss function is of the form

$L(\theta ,a)={\begin{cases}0,&{\text{if }}|a-\theta |<c,\\1,&{\text{otherwise}},\\\end{cases}}$

as $c$ goes to 0, the Bayes estimator approaches the MAP estimator, provided that the distribution of $\theta$ is unimodal. But generally a MAP estimator is not a Bayes estimator unless $\theta$ is discrete.

Example

Suppose that there is a sample $x_{1},x_{2},...,x_{n}$ of n independent and identically distributed observations coming from $x_{i}|\theta \backsim N(\theta ,\sigma ^{2})$ and a priori distribution of $\theta$ is given by $N(\mu _{0},\sigma _{0}^{2})$. We wish to find the MAP estimate of $\theta$. Note that the normal distribution is its own conjugate prior, so we will be able to find a closed-form solution analytically.

The MAP estimate of $\theta$ is

${\hat {\theta }}_{\mathrm {MAP} }={\underset {\theta }{\operatorname {arg\,max} }}\ f(x_{1},x_{2},...,x_{n}\mid \theta )\,g(\theta )\!$

where $f$ is a joint density function of all observations given the population mean equals to $\theta$ and $g$ is a density function of $N(\mu _{0},\sigma _{0}^{2})$.

Since the logarithm function itself is a monotonic function, the values that maximize the optimization objective will also maximize its logarithm.

Therefore, by taking the logarithm function to the optimization objective:

${\hat {\theta }}_{\mathrm {MAP} }{\displaystyle ={\underset {\theta }{\operatorname {arg\,max} }}\ log[f(x_{1},x_{2},...,x_{n}\mid \theta )\,g(\theta )]\!}$

Since $x_{i}$'s are independent and identically distributed normal random variables, the optimization objective can be written as follows:

${\displaystyle ={\underset {\theta }{\operatorname {arg\,max} }}\ \sum _{i=1}^{n}log[f(x_{i}\mid \theta )]+log[g(\theta )]\!}$

${\displaystyle ={\underset {\theta }{\operatorname {arg\,max} }}\ log{\biggl (}{\frac {n}{{\sqrt {2\pi }}\sigma }}{\biggr )}+\sum _{i=1}^{n}{\biggl (}-{\frac {1}{2}}\left({\frac {x_{i}-\theta }{\sigma }}\right)^{2}{\biggr )}+log{\biggl (}{\frac {1}{{\sqrt {2\pi }}\sigma _{0}}}{\biggr )}-{\frac {1}{2}}\left({\frac {\theta -\mu _{0}}{\sigma _{0}}}\right)^{2}\!}$

The first and third terms of the optimization objective above do not depend on $\theta$. Hence, they can be omitted as below:

${\displaystyle ={\underset {\theta }{\operatorname {arg\,min} }}\ \sum _{i=1}^{n}\left({\frac {x_{i}-\theta }{\sigma }}\right)^{2}+\left({\frac {\theta -\mu _{0}}{\sigma _{0}}}\right)^{2}\!}$

Note that the problem changed from a maximization problem to a minimization problem by negating the objective after omitting the first and third terms from the previous step. Since the final objective, ${\sum _{i=1}^{n}\left({\frac {x_{i}-\theta }{\sigma }}\right)^{2}+\left({\frac {\theta -\mu _{0}}{\sigma _{0}}}\right)^{2}\!}$ , is a convex function of $\theta$, it has a global minimum at some value $\theta$ such that

${{\partial \over \partial \theta }\sum _{i=1}^{n}\left({\frac {x_{i}-\theta }{\sigma }}\right)^{2}+\left({\frac {\theta -\mu _{0}}{\sigma _{0}}}\right)^{2}=0\!}$

Thus, by taking the partial derivative with respect to $\theta$, the MAP estimate of $\theta$ can be derived as

${\hat {\theta }}_{\mathrm {MAP} }={\frac {\sigma _{0}^{2}\,n}{\sigma _{0}^{2}\,n+\sigma ^{2}}}\left({\frac {1}{n}}\sum _{i=1}^{n}x_{i}\right)+{\frac {\sigma ^{2}}{\sigma _{0}^{2}\,n+\sigma ^{2}}}\,\mu _{0}={\frac {\sigma _{0}^{2}\left(\sum _{i=1}^{n}x_{i}\right)+\sigma ^{2}\,\mu _{0}}{\sigma _{0}^{2}\,n+\sigma ^{2}}}.$

which turns out to be a linear interpolation between the prior mean and the sample mean weighted by their respective covariances.

The case of $\sigma _{0}\to \infty$ is called a non-informative prior and leads to an ill-defined a priori probability distribution. In this case, ${\hat {\theta }}_{\mathrm {MAP} }\to {\hat {\theta }}_{MLE}.$

Computation

MAP estimates can be computed in several ways:

1. Analytically, when the mode(s) of the posterior distribution can be given in closed form. This is the case when conjugate priors are used.
2. Via numerical optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
3. Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.
4. Via a Monte Carlo method using simulated annealing

Criticism

While MAP estimation is a limit of Bayes estimators (under the 0-1 loss function), it is not very representative of Bayesian methods in general. This is because MAP estimates are point estimates, whereas Bayesian methods are characterized by the use of distributions to summarize data and draw inferences: thus, Bayesian methods tend to report the posterior mean or median instead, together with credible intervals. This is both because these estimators are optimal under squared-error and linear-error loss respectively - which are more representative of typical loss functions - and because the posterior distribution may not have a simple analytic form: in this case, the distribution can be simulated using Markov chain Monte Carlo techniques, while optimization to find its mode(s) may be difficult or impossible.

An example of a density of a bimodal distribution in which the highest mode is uncharacteristic of the majority of the distribution

In many types of models, such as mixture models, the posterior may be multi-modal. In such a case, the usual recommendation is that one should choose the highest mode: this is not always feasible (global optimization is a difficult problem), nor in some cases even possible (such as when identifiability issues arise). Furthermore, the highest mode may be uncharacteristic of the majority of the posterior.

Finally, unlike ML estimators, the MAP estimate is not invariant under reparameterization. Switching from one parameterization to another involves introducing a Jacobian that impacts on the location of the maximum.

As an example of the difference between Bayes estimators mentioned above (mean and median estimators) and using a MAP estimate, consider the case where there is a need to classify inputs $x$ as either positive or negative (for example, loans as risky or safe). Suppose there are just three possible hypotheses about the correct method of classification $h_{1}$, $h_{2}$ and $h_{3}$ with posteriors 0.4, 0.3 and 0.3 respectively. Suppose given a new instance, $x$, $h_{1}$ classifies it as positive, whereas the other two classify it as negative. Using the MAP estimate for the correct classifier $h_{1}$, $x$ is classified as positive, whereas the Bayes estimators would average over all hypotheses and classify $x$ as negative.