Lehmann–Scheffé theorem

In statistics, the Lehmann–Scheffé theorem provides sufficient conditions for the existence of a best unbiased estimator in a statistical model. The theorem states that any unbiased estimator for a quantity that depends on the data only through a complete, sufficient statistic is the unique uniformly minimum-variance unbiased estimator (UMVUE) of that quantity. The Lehmann–Scheffé theorem is named after Erich Leo Lehmann and Henry Scheffé, given their two early papers.

Introduction

Given a vector $X=(X_{1},X_{2},\dots ,X_{n})$ of random samples from a distribution $\mathbb {P} _{\theta }$ for some parameter $\theta \in \Theta$, the goal is to establish sufficient conditions for the existence of an UMVU estimator $T^{\ast }(X)$ for some quantity $g(\theta )$, that is, $\mathbb {E} _{\theta }[T^{\ast }(X)]=g(\theta )$ and for any unbiased estimator $S(X)$ it holds

$$\mathrm {Var} _{\theta }(T^{\ast }(X))\leq \mathrm {Var} _{\theta }(S(X)),\quad \forall \theta \in \Theta .$$

The Rao–Blackwell theorem already shows that, given a sufficient statistic $T$, the estimator $T^{\ast }(X)=\mathbb {E} [S(X)\mid T(X)]$ has a uniformly smaller variance than $S(X)$, but it does not guarantee that $T^{\ast }$ is already UMVU. This is where the Lehmann–Scheffé theorem comes in, if $T$ is also complete, that is, for any real-valued measurable function $\varphi$ it holds

$$\mathbb {E} _{\theta }[\varphi (T(X))]=0\implies \varphi (T(X))=0\quad \mathbb {P} _{\theta }{\text{-a.s.}}$$

Statement

As above, let $X=(X_{1},X_{2},\dots ,X_{n})$ be a vector of random samples from a distribution $\mathbb {P} _{\theta }$ for some parameter $\theta \in \Theta$ and $\Theta$ an arbitrary set.

Assume that there exists a complete, sufficient statistic $T$ for the family of distributions $(\mathbb {P} _{\theta })_{\theta \in \Theta }$. Then, the following two equivalent statements hold:

• There exists at most one measurable function $h$ such that $T^{\ast }=h(T(X))$ is unbiased for $g(\theta )$ and $\mathbb {E} _{\theta }[T^{\ast }(X)^{2}]<\infty$ for all $\theta \in \Theta$, in which case $T^{\ast }(X)$ is the unique UMVUE for $g(\theta )$.
• For any unbiased estimator $S(X)$, if it exists, with $\mathbb {E} _{\theta }[S(X)^{2}]<\infty$ for all $\theta \in \Theta$, the estimator $T^{\ast }(X)=\mathbb {E} _{\theta }[S(X)\mid T(X)]$ is the unique UMVUE for $g(\theta )$.

In fact, the theorem does not state that unbiased estimators exist in the first place. However, if they do, then there exists a unique square-integrable UMVUE. Moreover, the estimator $T^{\ast }(X)=\mathbb {E} _{\theta }[S(X)\mid T(X)]$ does neither depend on $\theta$, since $T$ is sufficient, nor on $S$, since $T$ is also complete.

Proof

In the following, the dependence of an estimator on the data $X$ will not be written out explicitly, i.e., we write $S$ instead of $S(X)$.

First of all, if there is no unbiased estimator for $g(\theta )$, then there is obviously no UMVUE, and if all unbiased estimator are not square-integrable, then their variances are infinity and the statement is trivial. Thus, we focus on the case where a square-integrable unbiased estimator exists.

Uniqueness of $h$: Let $S=h_{1}(T)$ and $R=h_{2}(T)$ be unbiased estimators of $g(\theta )$ for some measurable functions $h_{1}$ and $h_{2}$. For the expectation of the difference it holds$$\mathbb {E} _{\theta }[h_{1}(T)-h_{2}(T)]=g(\theta )-g(\theta )=0.$$Since $T$ is complete, this implies $h_{1}-h_{2}=0$. Thus, $T^{\ast }=h_{1}(T)=:h(T)$ is the unique unbiased estimator that is a function of $T$.

$h(T)$ is the UMVUE: Let $S$ be any square-integrable unbiased estimator and $S^{\ast }=\mathbb {E} _{\theta }[S\mid T]$. By the factorization lemma there exists measurable function ${\tilde {h}}$ such that $S^{\ast }={\tilde {h}}(T)$ and since $S^{\ast }$ is unbiased as well, by the above, it must hold $S^{\ast }={\tilde {h}}(T)=h(T)=T^{\ast }$. Thus, by the Rao–Blackwell theorem, it follows$$\mathrm {Var} _{\theta }(T^{\ast })=\mathrm {Var} _{\theta }(S^{\ast })=\mathrm {Var} _{\theta }(\mathbb {E} _{\theta }[S\mid T])\leq \mathrm {Var} _{\theta }(S),\quad \forall \theta \in \Theta .$$In other words, $T^{\ast }=h(T)$ is an UMVUE and according to the first part it is unique.

Application

The Lehmann–Scheffé theorem motivates two general methods to construct UMVU estimators for $g(\theta )$ in models which allow for a complete sufficient statistic $T$.

Method 1: Determining the function $h$

The UMVUE, if it exists, is the (unique) solution of the equation$$\mathbb {E} _{\theta }[h(T)]=g(\theta )$$for all $\theta \in \Theta$. If $h$ is, for example, a linear function, then solving this equation is fairly easy.

Method 2: Conditioning on an unbiased estimator

First, it suffices to find any unbiased estimator $S$ of $g(\theta )$, which is often easily feasible. The UMVUE can then be determined by evaluating the condition expectation $\mathbb {E} _{\theta }[S\mid T]$. Since the choice of $S$ is arbitrary, it is preferable to choose it such that the conditional expectation is as simple as possible.

Examples

Bernoulli distribution

Let $X_{1},\dots ,X_{n}$ be Bernoulli-distributed with probability $p\in (0,1)$. The joint probability mass function $f_{p}$ is of the form$$f_{p}(x_{1},\dots ,x_{n})=\prod _{i=1}^{n}p^{x_{i}}(1-p)^{1-x_{i}}=\left({\frac {p}{1-p}}\right)^{\sum _{i=1}^{n}x_{i}}(1-p)^{n},\quad x_{i}\in \{0,1\},$$thus, by the Fisher–Neyman factorization theorem, $T(x)=\sum _{i=1}^{n}x_{i}$ is a sufficient statistic. It is also complete: Let $\varphi$ be any measurable function such that $\mathbb {E} _{\theta }[\varphi (T(X))]=0$. Since $T$ is $\mathrm {Ber} (n,p)$-distributed, this means$$0=\mathbb {E} _{\theta }[\varphi (T(X))]=\sum _{k=0}^{n}\varphi (k){\binom {n}{k}}p^{k}(1-p)^{n-k}=(1-p)^{n}\sum _{k=0}^{n}\varphi (k){\binom {n}{k}}r^{k},$$with $r=p/(1-p)$. Since the right hand side is a polynomial in $r>0$ that is equal to zero, each coefficient must be zero as well, implying $\varphi (k)=0$ for all $k$.

For the estimation of the parameter $p$, it is now easy to see that the UMVUE is the sample mean$${\overline {X}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}={\frac {1}{n}}T(X),$$since it is unbiased and a function of $T(X)$.

Finding the UMVUE for the parameter $p(1-p)$ (that is, the variance of the distribution) is less obvious. According to method 1, we seek a function $h$ such that $$p(1-p)=\mathbb {E} _{p}[h(T(X))]=\sum _{k=0}^{n}h(k){\binom {n}{k}}p^{k}(1-p)^{n-k}.$$By defining $\rho =p/(1-p)$, the above rewrites as$$\sum _{k=0}^{n}h(k){\binom {n}{k}}\rho ^{k}=\rho (1+\rho )^{n-2}=\sum _{k=1}^{n-1}{\binom {n-1}{k-1}}\rho ^{k}.$$Comparing the coefficients shows that $h(k)=k(n-k)/(n(n-1))$, thus, the UMVUE is given by$$h(T(X))={\frac {n}{n-1}}{\overline {X}}(1-{\overline {X}}).$$Noting that in the Bernoulli model $X_{i}=X_{i}^{2}$ and that $\sum _{i=1}^{n}(X_{i}-{\overline {X}})^{2}=\sum _{i=1}^{n}X_{i}^{2}-n{\overline {X}}^{2}$, the UMVUE is, in fact, the unbiased sample variance $S_{n}^{2}$.

Uniform distribution

Let $X_{1},\dots ,X_{n}$ be uniformly distributed on the interval $(0,\theta )$ for some $\theta$ to be determined. The joint probability mass function $f_{p}$ is of the form$$f_{p}(x_{1},\dots ,x_{n})=\prod _{i=1}^{n}{\frac {1}{\theta }}1_{(0,\theta )}(x_{i})={\frac {1}{\theta ^{n}}}1_{[0,\infty )}\left(\min _{i=1,\dots ,n}x_{i}\right)1_{(-\infty ,\theta ]}\left(\max _{i=1,\dots ,n}x_{i}\right),\quad x\in \mathbb {R} ^{n},$$thus, by the Fisher–Neyman factorization theorem, $T(x)=\max _{i=1,\dots ,n}x_{i}$ is a sufficient statistic.

To see the completeness of $T$, the distribution of $T(X)$ is

$$\mathbb {P} _{\theta }(T(X)\leq t)=\mathbb {P} _{\theta }(X_{i}\leq t,\forall i=1,\dots ,n)=\prod _{i=1}^{n}\mathbb {P} _{\theta }(X_{i}\leq t)=\prod _{i=1}^{n}{\frac {t}{\theta }}={\frac {t^{n}}{\theta ^{n}}}.$$

Thus, for any measurable function $\varphi$ such that $\mathbb {E} _{\theta }[\varphi (T(X))]=0$ it holds for any $\theta >0$

$$0=\mathbb {E} _{\theta }[\varphi (T(X))]=\int _{0}^{\theta }\varphi (x){\frac {n}{\theta ^{n}}}x^{n-1}\mathrm {d} x\implies \int _{0}^{\theta }\varphi (x)x^{n-1}\mathrm {d} x=0,$$

and consequently for any $b>a>0$

$$\int _{a}^{b}\varphi (x)x^{n-1}\mathrm {d} x=\int _{0}^{b}\varphi (x)x^{n-1}\mathrm {d} x-\int _{0}^{a}\varphi (x)x^{n-1}\mathrm {d} x=0.$$

This implies that $\varphi =0$ (almost everywhere) on $[0,\infty )$. So $T$ is also complete. Its expectation is

$$\mathbb {E} _{\theta }[T(X)]=\int _{0}^{\theta }x{\frac {n}{\theta ^{n}}}x^{n-1}\mathrm {d} x={\frac {n}{\theta ^{n}}}{\frac {\theta ^{n+1}}{n+1}}={\frac {n}{n+1}}\theta .$$

Thus, the rescaled estimator

$$T^{\ast }(X)={\frac {n+1}{n}}T(X)={\frac {n+1}{n}}\max _{i=1,\dots ,n}X_{i}$$

is unbiased and since it is a function of $T$, it is the UMVUE for $\theta$.

Exponential distribution

Let $X_{1},\dots ,X_{n}$ be exponentially distributed with parameter $\lambda$ and suppose the parameter $g(\lambda )=\mathbb {P} _{\lambda }(X_{1}>t)$ should be estimated for some fixed $t>0$. A straightforward unbiased estimator for $g(\lambda )$ would be ${\widehat {g(\lambda )}}={\frac {1}{n}}\sum _{i=1}^{n}1_{\{X_{i}>t\}}$, however, it turns out to be sub-optimal.

Since the exponential distribution is an exponential family, it is known that its sufficient statistic ${\overline {X}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}$ is also complete. Moreover, the estimator $T=1_{\{X_{1}>t\}}$ is unbiased. Thus, according to method 2, the estimator $T^{\ast }=\mathbb {E} [T\mid {\overline {X}}]=\mathbb {P} (X_{1}>t\mid {\overline {X}})$ is the UMVUE. It remains to evaluate the conditional expectation.

First of all, $X_{1}/(n{\overline {X}})\sim \mathrm {Beta} (1,n-1)$, which is independent of $\lambda$ (also called ancillary). In fact, since $X_{1}$ and $S=\sum _{i=2}^{n}X_{i}$ are independent and $S\sim \Gamma (n-1,\lambda )$, the cumulative distribution function at $y\in (0,1)$ reads as

$${\begin{aligned}\mathbb {P} \left({\frac {X_{1}}{n{\overline {X}}}}\leq y\right)&=\mathbb {P} \left({\frac {X_{1}}{X_{1}+S}}\leq y\right)\\&=\int _{0}^{\infty }\int _{0}^{\infty }1_{\{x/(x+s)\leq y\}}\lambda e^{-\lambda x}{\frac {\lambda ^{n-1}}{(n-2)!}}s^{n-2}e^{-\lambda s}\mathrm {d} x\mathrm {d} s\\&=\int _{0}^{1}\int _{0}^{\infty }1_{\{u\leq y\}}{\frac {\lambda ^{n}}{(n-2)!}}r^{n-2}(1-u)^{n-2}e^{-\lambda r}r\mathrm {d} r\mathrm {d} u\\&=\int _{0}^{1}1_{\{u\leq y\}}(1-u)^{n-2}{\frac {1}{(n-2)!}}\underbrace {\int _{0}^{\infty }\lambda ^{n}r^{n-1}e^{-\lambda r}\mathrm {d} r} _{=(n-1)!}\mathrm {d} u\\&=\int _{0}^{y}(n-1)(1-u)^{n-2}\mathrm {d} u,\end{aligned}}$$

with the substitution $x=ru$ and $s=r(1-u)$. Thus, by Basu's theorem, $X_{1}/(n{\overline {X}})$ and ${\overline {X}}$ are independent and the conditional expectation simplifies to

$$\mathbb {P} (X_{1}>t\mid {\overline {X}}=x)=\mathbb {P} \left({\frac {X_{1}}{n{\overline {X}}}}>{\frac {t}{n{\overline {X}}}}\mid {\overline {X}}=x\right)=\mathbb {P} \left({\frac {X_{1}}{n{\overline {X}}}}>{\frac {t}{nx}}\right)=\left(1-{\frac {t}{nx}}\right)^{n-1}.$$

due to the above derivation. The constructed estimator

$$T^{\ast }=\left(1-{\frac {t}{n{\overline {X}}}}\right)^{n-1}$$

is unbiased and has a smaller variance than the naive estimator ${\widehat {g(\lambda )}}$ for all possible $\lambda >0$.

Counterexample with incomplete statistics

An example of an improvable Rao–Blackwell improvement, when using a minimal sufficient statistic that is not complete, was provided by Galili and Meilijson in 2016. Let $X_{1},\ldots ,X_{n}$ be a random sample from a scale-uniform distribution $X\sim U((1-k)\theta ,(1+k)\theta ),$ with unknown mean $\operatorname {E} [X]=\theta$ and known design parameter $k\in (0,1)$. In the search for "best" possible unbiased estimators for $\theta$, it is natural to consider $X_{1}$ as an initial (crude) unbiased estimator for $\theta$ and then try to improve it. Since $X_{1}$ is not a function of $T=\left(X_{(1)},X_{(n)}\right)$, the minimal sufficient statistic for $\theta$ (where $X_{(1)}=\min _{i}X_{i}$ and $X_{(n)}=\max _{i}X_{i}$), it may be improved using the Rao–Blackwell theorem as follows:

${\hat {\theta }}_{RB}=\operatorname {E} _{\theta }[X_{1}\mid X_{(1)},X_{(n)}]={\frac {X_{(1)}+X_{(n)}}{2}}.$

However, the following unbiased estimator can be shown to have lower variance:

${\hat {\theta }}_{LV}={\frac {1}{k^{2}{\frac {n-1}{n+1}}+1}}\cdot {\frac {(1-k)X_{(1)}+(1+k)X_{(n)}}{2}}.$

And in fact, it could be even further improved when using the following estimator:

${\hat {\theta }}_{\text{BAYES}}={\frac {n+1}{n}}\left[1-{\frac {{\frac {X_{(1)}(1+k)}{X_{(n)}(1-k)}}-1}{\left({\frac {X_{(1)}(1+k)}{X_{(n)}(1-k)}}\right)^{n+1}-1}}\right]{\frac {X_{(n)}}{1+k}}$

The model is a scale model. Optimal equivariant estimators can then be derived for loss functions that are invariant.

See also

• Completeness (statistics)
• Sufficient statistic
• Minimum-variance unbiased estimator
• Rao–Blackwell theorem

References

• Casella, George; Berger, Roger L. (2002). Statistical Inference (2nd ed.). Duxbury. ISBN 0-534-24312-6.
• Czado, Claudia; Schmidt, Thorsten (2011). Mathematische Statistik [Mathematical Statistics] (in German). Springer. ISBN 978-3-642-17261-8.
• Lehmann, Erich Leo; Casella, George (1998). Theory of Point Estimation (2nd ed.). New York: Springer. ISBN 0-387-98502-6.
• Shao, Jun (2003). Mathematical Statistics (2nd ed.). Springer. ISBN 978-0-387-95382-3.