In Bayes estimation, a risk is minimized to obtain the optimal estimate. The Bayes risk of estimate is defined as
where d is the observation, is a cost function and is the posterior distribution. First of all, we need to compute the posterior distribution from the prior and the likelihood. According to the Bayes rule, the posterior probability can be computed by using the following formulation
where is the prior probability of labelings f, is the conditional p.d.f. of the observations d, also called the likelihood function of f for d fixed, and is the density of d which is a constant when d is given.
Figure 1.6: Two choices of cost functions.
The cost function determines the cost of estimate f when the truth is . It is defined according to our preference. Two popular choices are the quadratic cost function
where is a distance between a and b, and the (0-1) cost function
where is any small constant. A plot of the two cost functions are shown in Fig.1.6.
The Bayes risk under the quadratic cost function measures the variance of the estimate
Letting , we obtain the minimal variance estimate
The above is the mean of the posterior probability.
For the cost function, the Bayes risk is
When , the above is approximated by
where is the volume of the space containing all points f for which . Minimizing the above is equivalent to maximizing the posterior probability. Therefore, the minimal risk estimate is
which is known as the MAP estimate. Because in (1.70) is a constant for a fixed d, is proportional to the joint distribution
Then the MAP estimate is equivalently found by
Obviously, when the prior distribution, , is flat, the MAP is equivalent to the maximum likelihood.