Example Problems on Estimators
ST702 Homework 5 on Estimators
7.45
Let X1,X2,…Xn be iid from a distribution with mean μ and variance σ2, and let S2 be the usual unbiased estimator of σ2. In Example 7.3.4 we saw that, under normality, the MLE has smaller MSE than S2. In this exercise we will explore variance estimators some more.
a
Show that, for any estimator of the form aS2, where a is some constant,
MSE(aS2)=E[aS2−σ2]2=a2Var(S2)+(a−1)2σ4 MSE(aS2)=E[W−θ]2=E[aS2−σ2]2=Var(aS2)−(Bias(aS2))2=a2Var(S2)−(E(aS2)−σ2)2=a2Var(S2)−(a2E(S2)2−2aE(S2)σ2+σ4)=a2Var(S2)−(a2(σ2)2−2aσ2σ2+σ4)unbiased=a2Var(S2)+(a−1)2σ4b
Show that
Var(S2)=1n(κ−n−3n−1)σ4where κ=E[X−μ]4/σ4 is the kurtosis.
V(S2)=V(1n−1∑(Xi−¯X)2)=1(n−1)2∑V((Xi−¯X)2)=1(n−1)2∑[E(((Xi−¯X)2)2)−E((Xi−¯X)2)2]=1(n−1)2∑[E((Xi−¯X)4)−E((Xi−¯X)2)2]=1(n−1)2∑κσ4−(σ2)2=1(n−1)2(nκσ4−nσ4)c
Show that under normality, the kurtosis is 3 and establish that, in this case, the estimator of the form aS2 with the minimum MSE is n−1n+1S2. (Lemma 3.6.5 may be helpful)
We can bring the σ4 inside the fraction to get
κ=E[(X−μσ)4].Notice that the inside function now follows a N(0,1) distribution. We can also make use of Lemma 3.6.5 which says
E[g(x)(x−μ)]=σ2E(g′(x)).Now we can take g(Z)=Z3. Then,
κ=E[Z4]=E[g(Z)(Z−0)]=σ2E[g′(Z)]=σ2E[3Z2]=3σ2E[Z2]=3σ2(V(Z)+E(Z)2)=3⋅1(1+0)=3.Now we can use our MSE equation from (a) and our variance equation from (b). We are trying to optimize with respect to a.
MSE=a21n(κ−n−3n−1)σ4+(a−1)2σ4∂MSE∂a=2a1n(κ−n−3n−1)σ4+2(a−1)σ4=0a=σ41n(κ−n−3n−1)σ4+σ4=σ41n(3−n−3n−1)σ4+σ4=11n(3−n−3n−1)+1=n−1n+1d
If normality is not assumed, show that MSE(aS2) is minimized at
a=n−1n+1+(κ−3)(n−1)nwhich is useless as is depends on a parameter.
From above
a=σ41n(κ−n−3n−1)σ4+σ4=11n(κ−n−3n−1)+1=n−1n+1+(κ−3)(n−1)ne
Show that
i
for distributions with κ>3, the optimal a will satisfy a<n−1n+1.
Taking κ>3 will make the second term in our denominator positive, which will make the denominator larger than n+1. So our overall expression will be less than n−1n+1.
ii
for distributions with κ<3, the optimal a will satisfy n−1n+1<a<1.
Similarly, making κ<3 will make the second term in our denominator negative, which will make the denominator smaller than n+1. So our overall expression will be greater than n−1n+1.
7.48
Suppose that Xi, i=1,…,n are iid Bernoulli(p).
a
Show that the variance of the MLE of p attains the Cramér-Rao Lower Bound.
The MLE is,
L(p|x)=∏f(xi|p)=∏pxi(1−p)1−xi=p∑xi(1−p)1−∑xiℓ(p|x)=∑xilog(p)+(n−∑xi)log(1−p)∂ℓ∂p=∑xi1p+−(n−∑xi)11−p0=∑xi1p+−(n−∑xi)11−p∑xi1p=(n−∑xi)11−p1−pp=n−∑xi∑xiˆp=¯X.The Cramér-Rao lower bound is,
Vθ(W(X))≥(∂∂θEθ(W(X)))2−Eθ(∂2∂θ2log(f(x|θ)))−Eθ(∂2∂θ2log(f(x|θ)))=−Eθ(∂∂θ∑xip−(n−∑xi)11−p)=−Eθ(−∑xip2+−(n−∑xi)(1−p)2)=1p2∑E(xi)+E(n=∑xi(1−p)2)=np2p+n−np(1−p)2=np+n(1−p)(1−p)2CRLB=1np+n(1−p)(1−p)2=p(1−p)n.Now we can find the variance of our estimator.
V(ˆp)=V(∑xin)=1n2∑V(xi)=1n2np(1−p)=p(1−p)n=CRLBb
For n≥4, show that the product X1X2X3X4 is an unbiased estimator of p4. and use this fact to find the best unbiased estimator of p4.
E[X1X2X3X4]=E(X1)E(X2)E(X3)E(X4)independence=p4.Also ∑ni=1xi is sufficient for p by the Factorization Theorem.
f(X|p)=∏pxi(1−p)1−xi=p∑xi(1−p)1−∑xiWe can use the Rao-Blackwell theorem to find a UMVUE.
ϕ(T)=E(X1X2X3X4|n∑i=1Xi=t)=P(X1,X2,X3,X4=1∧∑ni=1Xi=t)P(∑ni=1Xi=t)=P(X1,X2,X3,X4=1∧∑ni=5Xi=t−4)P(∑ni=1Xi=t)=P(X1,X2,X3,X4=1)P(∑ni=5Xi=t−4)P(∑ni=1Xi=t)independence=p4⋅(n−4t−4)pt−4(1−p)n−4−(t−4)(nt)pt(1−p)n−t=(n−4t−4)(nt)7.49
Let X1,…,Xn be iid exponential(λ).
a
Find an unbiased estimator of λ based only on Y=minX1,…,Xn.
FY=P(X<Xi)=[1−F(X)]nfy=nf(x)(1−F(X))n−1=n1λe−1λx(1−(1−e−1λx))n−1=n1λe−n1λx.So Y∼Exp(λ/b). So nY is unbiased.
E(nY)=nE(Y)=nλ/n=λb
Find a better estimator than the one in part (a). Prove that it is better.
Since it’s an exponential family, ∑xi is sufficient and complete. Notice that ¯X is a function of ∑xi and is unbiased.
E(¯X)=1n∑E(xi)=1nnλ=λBy Lehmann-Scheffe, ¯X is UMVUE, so it is better than the estimator in (a).
c
The following data are high-stress failure times (in hours) of Kevlar/epoxy spherical vessels used in a sustained pressure environment on the space shuttle:
50.1,70.1,137.0,166.9,170.5,152.8,80.5,123.5,112.6,148.5,160.0,125.4.Failure times are often modeled with the exponential distribution. Estimate the mean failure time using the estimators from parts (a) and (b).
From (a) ˆλ=50.1⋅12=601.2. For (b) we can take the average ˆλ=124.825.
4
Consider a random sample X1,X2,…Xn from the density f(x|θ)=exp[−(x−θ)], θ<x<∞, and 0 elsewhere. Show that the first order statistic X(1)=minXi is a complete sufficient statistic for θ, and find the UMVUE of θ.
f(X|θ)=∏e−(x−θ)I[θ<x<∞]=e−∑xi−nθI[θ<X(1)]=e−∑xienθI[θ<X(1)]By the factorization theorem X(1) is sufficient. Now we can show that it is complete. First, we need its distribution.
fX(1)=n⋅fx(1−FX)n−1=ne−(x−θ)[1−∫xθe−(t−θ)dt]n−1=ne−(x−θ)[1−(1−e−x+θ)]n−1=ne−(x−θ)[e−x+θ]n−1=nen(−x+θ)Now we can show completeness.
∫∞θg(x)nen(θ−t)dx=nenθ∫∞θg(x)e−ntdxThis is only 0 if g(x) is 0. Now we can look at the expectation of the minimum order statistic.
E(X(1))=∫∞θxne(−x+θ)ndx=n(1+nθ)n2=1n+θSo X(1)−1n in unbiased.