Gauss - Markov Theorem / BLUE Properties

Given the assumptions of the classical linear regression model, the least-squares estimators, in the class of unbiased linear estimators, have minimum variance, that is, they are BLUE. In other words, the Gauss-Markov theorem holds the properties of Best Linear Unbiased Estimators.

Following are some of the assumptions that should be taken into consideration for the mathematical derivation of the Gauss-Markov Theorem :

Properties of $k_i$ ;

1. $\sum_{}^{}{k_i}=\; 0$

2. $\sum_{}^{}{k_ix_i}=\; \sum_{}^{}{k_iX_i}\; =\; 1$

3. $\sum_{}^{}{k_i^{2}}=\; \frac{1}{\sum_{}^{}{x_i^{2}}}$

Assumptions of $u_i$ ;

1. $E\left( u_i \right)\; =\; 0$

2. $Var\left( u_i \right)\; = E\left( u_i^{2} \right)\; =\; \sigma _{u}^{2}\; =\;Constant$

3. $u_i\; \sim \; N\left( 0, \;\sigma _{u}^{2} \right)$

4. $E\left( u_i,u_j \right)\; =\; 0$

5. $E\left( u_i,\; X_i \right)\; =\; X_i E\left(u_i \right)\; =\; 0$

Properties of BLUE (Best Linear Unbiased Estimators)

Lets take the regression line $Y\; =\; \alpha \; +\; \beta X_{i}\; +\; u_{i}$

1. Linearity i.e., $\hat{\beta}$

Lets take the regression line $Y\; =\; \alpha \; +\; \beta X_i\; +\; u_{i}$

As we know,

$\hat{\beta} = \frac{\sum_{}^{}{x_{i}y_{i}}}{\sum_{}^{}{x_{i^{}}^{2}}}$ ,

$=\frac{\sum_{}^{}{\left( X- \bar{X} \right)\left( Y-\bar{Y} \right)}}{\sum_{}^{}{x_{i}^{2}}}$ ,

$= \frac{\sum_{}^{}{\left( X- \bar{X} \right)Y\; -\; \sum_{}^{}{\left( X-\bar{X} \right)\bar{Y}}}}{\sum_{}^{}{x_{i}^{2}}}$ ,

$=\; \frac{\sum_{}^{}{x_i\; Yi}}{\sum_{}^{}{x_{i}^{2}}}$ ,

$=\; \sum_{}^{}{k_{i}Y_{i}}$ because $[\frac{x_{i}}{\sum_{}^{}{x_{i}^{2}}}=\; k_{i}]$

2. Unbiasedness i.e., $\hat\beta$ is unbiased,

From Linearity,

$\hat{\beta}=\; \sum_{}^{}{k_{i}Y_{i}}$ ,

$=\; \sum_{}^{}{k_{i}\; \left(\alpha \; +\; \beta X_{i}\; +\; u_{i} \right)}$ ,

$=\; \alpha \sum_{}^{}{k_{i}}+\; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}}$ ,

$=\; \; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}}$ , because $[\sum_{}^{}{k_i}=\; 0]$ ,

Taking expectations both sides,

$E\left( \hat{\beta} \right)\; =\; \beta$ , because $[\sum_{}^{}{k_ix_i}=\; 1 \;,\; E\left( u_i \right)\; =\; 0 ]$ ,

3. Best i.e, $\hat\beta$ is best or $\hat\beta$ have minimum variance

As we know,

$Var\left(\hat \beta \right)\; =\; E\left\{ \hat \beta \; -\beta \right\}^{2}$ ,

$=\;E\; \left\{ \beta \; +\; \sum_{}^{}{k_{i}u_{i}\; +\; \beta } \right\}^{2}$ ,

$=\;E\; \left\{ \sum_{}^{}{k_{i}u_{i}} \right\}^{2}$ ,

$=\; E\left\{ k_{1}^{2}u_{1}^{2}+k_{2}^{2}u_{2}^{2}+k_{3}^{2}u_{3}^{2}........... \right\} + 2E\left\{ k_{1}k_{2}u_{1}u_{2}+ k_{2}k_{3}u_{2}u_{3} +........ \right\}$ ,

$= E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\}\; +\; 2E\left\{ k_{i}k_{j}u_{i}u_{j} \right\}$ ,

$=\; E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\}$ ,

$=\; \sum_{}^{}{k_{i}^{2}\; E\left( u_{i}^{2} \right)}$ , beacause $[E\left( k_{i}k_{j}u_{i}u_{j} \right)\; =\; 0]$ ,

$= \sigma _{ui}^{2} \;\sum_{}^{}{k_{i}^{2}\; }$ ,

Now suppose there is another estimator,

$var\left( \beta^\ast \right)\; =\; \sum_{}^{}{u_{i}^{2}}\sigma _{ui}^{2}$ ,

Where $k_{i}\; \neq \; w_{i}$ ,

Let $w_{i}\; =\; k_{i}\; +\; c_{i}$ ,

So,

$Var\; \left( \beta^\ast \right)\; =\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)^{2}\; \sigma _{ui}^{2}}$ ,

$=\; \sigma _{ui}^{2}\; \sum_{}^{}{\left\{ k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right\}}$ ,

$=\; \sigma _{ui}^{2}\; \sum_{}^{}{k_{i}^{2}}+\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}}$ , As $[\sum k_{i}c_{i} = 0]$ ,

$=\; Var\left( \hat{\beta} \right)\; +\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}}$ ,

So, $Var\; \left( \hat\beta \right)\; < Var\left( \beta^\ast \right)$

Now for $\alpha$ ;

1. Linearity i.e.; $\alpha$ is linear,

As we know,

$\hat\alpha \; =\; \bar{Y}\; -\; \hat\beta \bar{X}$ ,

$=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\hat\beta \bar{X}$ ,

$=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\; \sum_{}^{}{k_{i}Y_{i}\bar{X}}$ ,

$=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i$ ,

So, $\hat\alpha$ is linear.

2. Unbiasedness i.e.; $\hat\alpha$ is unbiased,

From linearity,

$\hat\alpha=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i$ ,

$=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\; \bar{X} \right\}\; \left( \alpha \; +\; \beta X_{i}\; +\; u_{i} \right)}$ ,

$=\; \sum_{}^{}{\left\{ \frac{\alpha }{n}\; -\; \alpha k_{i}\bar{X}\; +\; \frac{\beta X_{i}}{n}\; -\; \beta X_{i}k_{i}\bar{X}\; +\; \frac{u_{i}}{n}\; -\; u_{i}k_{i}\bar{X} \right\}}$ ,

$=\; \frac{n\alpha }{\alpha }\; -\; \alpha \bar{X}\sum_{}^{}{k_{i}\; +\; \beta \; \sum_{}^{}{\frac{X_{i}}{n}\; -\; \beta \bar{X}\sum_{}^{}{k_{i}X_{i}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}-\sum_{}^{}{k_{i}u_{i}\bar{X}}}}}\;$ ,

$=\; \alpha \; +\; \beta \bar{X}\; -\; \beta \bar{X}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}\; -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}$ ,

Now taking expectations both sides,

$E\left( \hat\alpha \right)\; =\; \alpha \; +\; E\left( ui \right)\; \frac{1}{n}$ ,

$E\left( \hat\alpha \right)\; =\; \alpha \;$ ,

3. Best i.e.; $\hat\alpha$ is best or $\hat\alpha$ have minimum variance,

From unbiased,

$\hat\alpha =\; \alpha \; +\;\; \frac{\sum{u_i}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}$ ,

We know,

$var\left( \alpha \right)\; =\; E\left\{ \hat\alpha -\alpha \right\}^{2}$ ,

$=\; E\left\{ \alpha +\frac{\sum_{}^{}{u_{i}}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}-\alpha \right\}^2$ ,

$=\; E\left\{ \sum_{}^{}{\left( \frac{1}{n}\; -\; k_{i}\bar{X} \right)^{2}}u_{i}^{2} \right\}$ ,

$=\; \sigma _{ui}^{2}\left\{ \sum_{}^{}{\left( \frac{1}{n^{2}}\; +\; k_{i}^{2}\bar{X}\; -2\; \frac{1}{n}ki\bar{X} \right)^{}} \right\}$ ,

$=\; \sigma _{ui}^{2}\left\{ \frac{n}{n^{2}}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}}-\; \frac{2}{n}\bar{X}\sum_{}^{}{ki} \right\}$ ,

$=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}} \right\}$ , As $[\sum_{}^{}{k_i}=\; 0 ]$ ,

Lets take another estimator $\alpha^\ast$ ,

$Var\left( \alpha^\ast \right)\; =\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{w_{i}^{2}} \right\}$ ,

Where $k_{i}\; \neq \; w_{i}$ and let $w_i=k_i\;+\;c_i$ ,

Now,

$Var\left( \beta \right)\; =\; \sigma _{ui}^{2}\; \left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)} \right\}$ ,

$=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right)} \right\}$ ,

$=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{k_{i}^{2}\; +\; \bar{X}\; \sum_{}^{}{c_{i}^{2}}} \right\}$ , As $[\sum k_{i}c_{i} = 0]$ ,