Chapman–Robbins bound

In statistics, the Chapman–Robbins bound or Hammersley–Chapman–Robbins bound is a lower bound on the variance of estimators of a deterministic parameter. It is a generalization of the Cramér–Rao bound; compared to the Cramér–Rao bound, it is both tighter and applicable to a wider range of problems. However, it is usually more difficult to compute.

The bound was independently discovered by John Hammersley in 1950,[1] and by Douglas Chapman and Herbert Robbins in 1951.[2]

Statement

edit

Let   be the set of parameters for a family of probability distributions   on  .

For any two  , let   be the  -divergence from   to  . Then:

Theorem — Given any scalar random variable  , and any two  , we have  .

A generalization to the multivariable case is:[3]

Theorem — Given any multivariate random variable  , and any  ,  

Proof

edit

By the variational representation of chi-squared divergence:[3]  Plug in  , to obtain:  Switch the denominator and the left side and take supremum over   to obtain the single-variate case. For the multivariate case, we define   for any  . Then plug in   in the variational representation to obtain:  Take supremum over  , using the linear algebra fact that  , we obtain the multivariate case.

Relation to Cramér–Rao bound

edit

Usually,   is the sample space of   independent draws of a  -valued random variable   with distribution   from a by   parameterized family of probability distributions,   is its  -fold product measure, and   is an estimator of  . Then, for  , the expression inside the supremum in the Chapman–Robbins bound converges to the Cramér–Rao bound of   when  , assuming the regularity conditions of the Cramér–Rao bound hold. This implies that, when both bounds exist, the Chapman–Robbins version is always at least as tight as the Cramér–Rao bound; in many cases, it is substantially tighter.

The Chapman–Robbins bound also holds under much weaker regularity conditions. For example, no assumption is made regarding differentiability of the probability density function p(x; θ) of  . When p(x; θ) is non-differentiable, the Fisher information is not defined, and hence the Cramér–Rao bound does not exist.

See also

edit

References

edit
  1. ^ Hammersley, J. M. (1950), "On estimating restricted parameters", Journal of the Royal Statistical Society, Series B, 12 (2): 192–240, JSTOR 2983981, MR 0040631
  2. ^ Chapman, D. G.; Robbins, H. (1951), "Minimum variance estimation without regularity assumptions", Annals of Mathematical Statistics, 22 (4): 581–586, doi:10.1214/aoms/1177729548, JSTOR 2236927, MR 0044084
  3. ^ a b Polyanskiy, Yury (2017). "Lecture notes on information theory, chapter 29, ECE563 (UIUC)" (PDF). Lecture notes on information theory. Archived (PDF) from the original on 2022-05-24. Retrieved 2022-05-24.

Further reading

edit
  • Lehmann, E. L.; Casella, G. (1998), Theory of Point Estimation (2nd ed.), Springer, pp. 113–114, ISBN 0-387-98502-6