Chapman–Robbins bound

In statistics, the Chapman–Robbins bound or Hammersley–Chapman–Robbins bound is a lower bound on the variance of estimators of a deterministic parameter. It is a generalization of the Cramér–Rao bound; compared to the Cramér–Rao bound, it is both tighter and applicable to a wider range of problems. However, it is usually more difficult to compute. The bound was independently discovered by John Hammersley in 1950, and by Douglas Chapman and Herbert Robbins in 1951.

Statement

Let $\mathrm{\Theta }$ be the set of parameters for a family of probability distributions $\{{\mu }_{\theta }:\theta \in \mathrm{\Theta }\}$ on $\mathrm{\Omega }$.

For any two $\theta ,{\theta }^{\prime }\in \mathrm{\Theta }$, let ${\chi }^2({\mu }_{{\theta }^{\prime }};{\mu }_{\theta })$ be the ${\chi }^2$-divergence from ${\mu }_{\theta }$ to ${\mu }_{{\theta }^{\prime }}$. Then:

Theorem — Given any scalar random variable $\hat{g}:\mathrm{\Omega }\to \mathbb{ℝ}$, and any two $\theta ,{\theta }^{\prime }\in \mathrm{\Theta }$, we have ${\mathrm{Var}}_{\theta }[\hat{g}]\ge \underset{{\theta }^{\prime }\ne \theta \in \mathrm{\Theta }}{\sup }\frac{(E_{{\theta }^{\prime }}[\hat{g}]-E_{\theta }[\hat{g}]{)}^2}{{\chi }^2({\mu }_{{\theta }^{\prime }};{\mu }_{\theta })}$.

A generalization to the multivariable case is:

Theorem — Given any multivariate random variable $\hat{g}:\mathrm{\Omega }\to {\mathbb{ℝ}}^m$, and any $\theta ,{\theta }^{\prime }\in \mathrm{\Theta }$, ${\chi }^2({\mu }_{{\theta }^{\prime }};{\mu }_{\theta })\ge (E_{{\theta }^{\prime }}[\hat{g}]-E_{\theta }[\hat{g}]{)}^T{\mathrm{Cov}}_{\theta }[\hat{g}{]}^{-1}(E_{{\theta }^{\prime }}[\hat{g}]-E_{\theta }[\hat{g}])$

Proof

By the variational representation of chi-squared divergence:$${\chi }^2(P;Q)=\underset{g}{\sup }\frac{(E_P[g]-E_Q[g]{)}^2}{{\mathrm{Var}}_Q[g]}$$ Plug in $g=\hat{g},P={\mu }_{{\theta }^{\prime }},Q={\mu }_{\theta }$, to obtain: $${\chi }^2({\mu }_{{\theta }^{\prime }};{\mu }_{\theta })\ge \frac{(E_{{\theta }^{\prime }}[\hat{g}]-E_{\theta }[\hat{g}]{)}^2}{{\mathrm{Var}}_{\theta }[\hat{g}]}$$Switch the denominator and the left side and take supremum over ${\theta }^{\prime }$ to obtain the single-variate case. For the multivariate case, we define $h={\sum }_{i=1}^m{v}_i{\hat{g}}_i$ for any $v\ne 0\in {\mathbb{ℝ}}^m$. Then plug in $g=h$ in the variational representation to obtain: $${\chi }^2({\mu }_{{\theta }^{\prime }};{\mu }_{\theta })\ge \frac{(E_{{\theta }^{\prime }}[h]-E_{\theta }[h]{)}^2}{{\mathrm{Var}}_{\theta }[h]}=\frac{⟨v,E_{{\theta }^{\prime }}[\hat{g}]-E_{\theta }[\hat{g}]{⟩}^2}{v^T{\mathrm{Cov}}_{\theta }[\hat{g}]v}$$Take supremum over $v\ne 0\in {\mathbb{ℝ}}^m$, using the linear algebra fact that $\underset{v\ne 0}{\sup }\frac{v^{T}w{w}^{T}v}{v^{T}Mv}=w^T{M}^{-1}w$, we obtain the multivariate case.

Relation to Cramér–Rao bound

Usually, $\mathrm{\Omega }={{𝒳}}^n$ is the sample space of $n$ independent draws of a ${𝒳}$-valued random variable $X$ with distribution ${\lambda }_{\theta }$ from a by $\theta \in \mathrm{\Theta }\subseteq {\mathbb{ℝ}}^m$ parameterized family of probability distributions, ${\mu }_{\theta }={\lambda }_{\theta }^{\otimes n}$ is its $n$-fold product measure, and $\hat{g}:{{𝒳}}^n\to \mathrm{\Theta }$ is an estimator of $\theta$. Then, for $m=1$, the expression inside the supremum in the Chapman–Robbins bound converges to the Cramér–Rao bound of $\hat{g}$ when ${\theta }^{\prime }\to \theta$, assuming the regularity conditions of the Cramér–Rao bound hold. This implies that, when both bounds exist, the Chapman–Robbins version is always at least as tight as the Cramér–Rao bound; in many cases, it is substantially tighter.

The Chapman–Robbins bound also holds under much weaker regularity conditions. For example, no assumption is made regarding differentiability of the probability density function p(x; θ) of ${\lambda }_{\theta }$. When p(x; θ) is non-differentiable, the Fisher information is not defined, and hence the Cramér–Rao bound does not exist.

See also

• Cramér–Rao bound
• Estimation theory

Further reading

• Lehmann, E. L.; Casella, G. (1998), Theory of Point Estimation (2nd ed.), Springer, pp. 113–114, ISBN 0-387-98502-6