Hammersley–Clifford theorem

The Hammersley–Clifford theorem is a result in probability theory, mathematical statistics and statistical mechanics that gives necessary and sufficient conditions under which a strictly positive probability distribution (of events in a probability space)[clarification needed] can be represented as events generated by a Markov network (also known as a Markov random field). It is the fundamental theorem of random fields.[1] It states that a probability distribution that has a strictly positive mass or density satisfies one of the Markov properties with respect to an undirected graph G if and only if it is a Gibbs random field, that is, its density can be factorized over the cliques (or complete subgraphs) of the graph.

The relationship between Markov and Gibbs random fields was initiated by Roland Dobrushin[2] and Frank Spitzer[3] in the context of statistical mechanics. The theorem is named after John Hammersley and Peter Clifford, who proved the equivalence in an unpublished paper in 1971.[4][5] Simpler proofs using the inclusion–exclusion principle were given independently by Geoffrey Grimmett,[6] Preston[7] and Sherman[8] in 1973, with a further proof by Julian Besag in 1974.[9]

Proof OutlineEdit

 
A simple Markov network for demonstrating that any Gibbs random field satisfies every Markov property.

It is a trivial matter to show that a Gibbs random field satisfies every Markov property. As an example of this fact, see the following:

In the image to the right, a Gibbs random field over the provided graph has the form  . If variables   and   are fixed, then the global Markov property requires that:   (see conditional independence), since   forms a barrier between   and  .

With   and   constant,   where   and  . This implies that  .

To establish that every positive probability distribution that satisfies the local Markov property is also a Gibbs random field, the following lemma, which provides a means for combining different factorizations, needs to be proven:

 
Lemma 1 provides a means for combining factorizations as shown in this diagram. Note that in this image, the overlap between sets is ignored.

Lemma 1

Let   denote the set of all random variables under consideration, and let   and   denote arbitrary sets of variables. (Here, given an arbitrary set of variables  ,   will also denote an arbitrary assignment to the variables from  .)

If

 

for functions   and  , then there exist functions   and   such that

 

In other words,   provides a template for further factorization of  .

Proof of Lemma 1

In order to use   as a template to further factorize  , all variables outside of   need to be fixed. To this end, let   be an arbitrary fixed assignment to the variables from   (the variables not in  ). For an arbitrary set of variables  , let   denote the assignment   restricted to the variables from   (the variables from  , excluding the variables from  ).

Moreover, to factorize only  , the other factors   need to be rendered moot for the variables from  . To do this, the factorization

 

will be re-expressed as

 

For each  :   is   where all variables outside of   have been fixed to the values prescribed by  .

Let   and   for each   so

 

What is most important is that   when the values assigned to   do not conflict with the values prescribed by  , making   "disappear" when all variables not in   are fixed to the values from  .

Fixing all variables not in   to the values from   gives

 

Since  ,

 

Letting   gives:

  which finally gives:

 

 
The clique formed by vertices  ,  , and  , is the intersection of  ,  , and  .

Lemma 1 provides a means of combining two different factorizations of  . The local Markov property implies that for any random variable  , that there exists factors   and   such that:

 

where   are the neighbors of node  . Applying Lemma 1 repeatedly eventually factors   into a product of clique potentials (see the image on the right).

End of Proof

See alsoEdit

NotesEdit

  1. ^ Lafferty, John D.; Mccallum, Andrew (2001). "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data". ICML. Retrieved 14 December 2014. by the fundamental theorem of random fields (Hammersley & Clifford, 1971)
  2. ^ Dobrushin, P. L. (1968), "The Description of a Random Field by Means of Conditional Probabilities and Conditions of Its Regularity", Theory of Probability and Its Applications, 13 (2): 197–224, doi:10.1137/1113026
  3. ^ Spitzer, Frank (1971), "Markov Random Fields and Gibbs Ensembles", The American Mathematical Monthly, 78 (2): 142–154, doi:10.2307/2317621, JSTOR 2317621
  4. ^ Hammersley, J. M.; Clifford, P. (1971), Markov fields on finite graphs and lattices (PDF)
  5. ^ Clifford, P. (1990), "Markov random fields in statistics", in Grimmett, G. R.; Welsh, D. J. A. (eds.), Disorder in Physical Systems: A Volume in Honour of John M. Hammersley, Oxford University Press, pp. 19–32, ISBN 978-0-19-853215-6, MR 1064553, retrieved 2009-05-04
  6. ^ Grimmett, G. R. (1973), "A theorem about random fields", Bulletin of the London Mathematical Society, 5 (1): 81–84, CiteSeerX 10.1.1.318.3375, doi:10.1112/blms/5.1.81, MR 0329039
  7. ^ Preston, C. J. (1973), "Generalized Gibbs states and Markov random fields", Advances in Applied Probability, 5 (2): 242–261, doi:10.2307/1426035, JSTOR 1426035, MR 0405645
  8. ^ Sherman, S. (1973), "Markov random fields and Gibbs random fields", Israel Journal of Mathematics, 14 (1): 92–103, doi:10.1007/BF02761538, MR 0321185
  9. ^ Besag, J. (1974), "Spatial interaction and the statistical analysis of lattice systems", Journal of the Royal Statistical Society, Series B, 36 (2): 192–236, JSTOR 2984812, MR 0373208

Further readingEdit