4

The question is: $x_i = \alpha + \omega_i, $ for $i = 1, \ldots, n.$

where $\alpha$ is a non-zero constant, but unknown, parameter to be estimated, and $\omega_i$ are uncorrelated, zero_mean, Gaussian random variable with known variance $\sigma_i^2$. Note that $\sigma_i^2$ and $\sigma_j^2$, for $i \neq j$, may be distinct. We wish to estimate $\alpha$ from a weighted sum of $x_i$, i.e.

$$\hat{\alpha} = \sum^n_{i=1}b_ix_i$$

Determine $b_i$, $i= 1, \ldots, n$, such that $\hat{\alpha}$ is unbiased and the variance of $\hat{\alpha}$ is as small as possible.

I have tried to use the unbiased condition and get that: $\sum_{i=1}^nb_i = 1$

I don't know how to use the variance of $\hat{\alpha}$ is as small as possible condition.

Michael Hardy
  • 1
user2262504
  • 954
  • I posted an answer that you'll understand if you know about Lagrange multipliers, one of the standard second-year calculus topics. For that answer you don't need to know about the Gauss-Markov theorem. Another person posted an answer that you'll understand if you know the Gauss-Markov theorem, but for that you don't need Lagrange multipliers. So maybe you'll find one of them useful and not the other, and which is which depends on your background. – Michael Hardy Nov 09 '13 at 21:42

2 Answers2

2

The weights should be proportional to the reciprocals of the variances: $$ b_k = \frac{1/\sigma_k^2}{\sum_{i=1}^n 1/\sigma_i^2}.\tag1 $$ This can be shown with Lagrange multipliers.

The variance of $\sum_{i=1}^n b_i x_i$ is $\sum_{i=1}^n b_i^2\sigma_i^2$. The problem is to minimize that subject to the constraint $\sum_{i=1}^n b_i=1$. The $i$th component of the gradient of the thing to be minimized is $2b_i\sigma_i^2$. The vector whose components are those has to be a scalar multiple of the gradient of the function $\sum_{i=1}^n b_i$. Notice that that happens with the weights in $(1)$.

Michael Hardy
  • 1
  • how do you get the variance, can you show the process explicitly? – user2262504 Nov 10 '13 at 00:52
  • $\operatorname{var}\left(\sum_{i=1}^n b_i x_i\right)$ $=\sum_{i=1}^n \operatorname{var}(b_i x_i)$ because of uncorrelatedness. Then, $\operatorname{var}(b_i x_i) = b_i^2 \operatorname{var}(x_i) = b_i^2 \sigma_i^2$. – Michael Hardy Nov 10 '13 at 05:28
  • The sign of using Lagrange Multipliers is maximise and minimise problem with constrains, right? So next time I will try to use it when I encounter such situation. Thank you for your patient explain. – user2262504 Nov 11 '13 at 00:05
1

For the unbiasedness, we have

$$ E\left(\hat{\alpha}\right)=E\left(\sum_{i=1}^nb_ix_i\right)=E\left(\sum_{i=1}^nb_i(\alpha+\omega_i)\right)=\alpha\sum_{i=1}^nb_i + E\left(\sum_{i=1}^nb_i\omega_i\right)=\alpha\sum_{i=1}^nb_i $$ and we get that $\sum_{i=1}^nb_i=1$ as you say.

Now, what follows is to simply make this homoscedastic so that we can use the Gauss-Markov theorem. Divide through by $\sigma_i$:

$$ \frac{x_i}{\sigma_i}=\frac{\alpha}{\sigma_i}+\frac{\omega_i}{\sigma_i}\Rightarrow x_i^*=\alpha\frac{1}{\sigma_i}+\omega_i^* $$ where $\omega_i^*\sim N(0, 1)$ (stars indicate variance adjusted). This satisfies the usual OLS conditions, so by the Gauss-Markov theorem OLS is efficient and unbiased. The estimator then is:

$$ \hat{\alpha}=\arg\min_{a}\sum_{i=1}^n(x_i^*-a\frac{1}{\sigma_i})^2\Rightarrow-2\sum_{i=1}^n\frac{(x_i^*-a\frac{1}{\sigma_i})}{\sigma_i}=0\\ \sum_{i=1}^n\frac{x_i}{\sigma_i^2}=\sum_{i=1}^n\frac{a}{\sigma^2_i}\\ \sum_{i=1}^n\frac{x_i}{\sigma^2_i}=a\sum_{i=1}^n\frac{1}{\sigma^2_i}\\ \frac{\sum_{i=1}^n\frac{x_i}{\sigma^2_i}}{\sum_{i=1}^n\frac{1}{\sigma^2_i}}=a $$ so the weights are $$ b_i=\frac{\frac{1}{\sigma^2_i}}{\sum_{i=1}^n\frac{1}{\sigma^2_i}}. $$

hejseb
  • 4,745
  • 1
  • 17
  • 35
  • You have a "$+$" where you need to multiply. – Michael Hardy Nov 09 '13 at 21:38
  • @MichaelHardy Thanks (you mean for the expectation, right?). I'm a little confused though, why are our answers different? – hejseb Nov 09 '13 at 22:27
  • Actually I hadn't noticed that they were different. I look at yours more closely now. – Michael Hardy Nov 10 '13 at 05:25
  • The first dubious-looking thing I see is where you say $\alpha/\sigma_i = \alpha^$ instead of $\alpha/\sigma_i = \alpha_i^$. Not all the $\alpha$s are equal. ${}\qquad{}$ – Michael Hardy Nov 10 '13 at 05:32
  • . . . and maybe that's the whole problem---I'm not sure yet. At any rate, the fact that the variance of the estimator is minimized by making the weights proportional to the reciprocals of the variances---not the reciprocals of the standard deviations---is a fairly standard result. – Michael Hardy Nov 10 '13 at 05:37
  • @MichaelHardy That is, of course, a correct observation. The subsequent differentiation is probably as a result a bit off then. I'll take a closer look again later. – hejseb Nov 10 '13 at 08:28
  • @MichaelHardy I fixed it now, it was what you pointed out that caused the error. Thanks for the help! – hejseb Nov 10 '13 at 08:36