Gauss Markov Theorem [BLUE Properties]

Given the assumptions of the classical linear regression model, the least-squares estimators, in the class of unbiased linear estimators, have minimum variance, that is, they are BLUE. In other words Gauss-Markov theorem holds the properties of Best Linear Unbiased Estimators. Following are some of the assumptions which should be taken into consideration for the mathematical derivation of the Gauss-Markov Theorem :

Properties of k_i;

1. \sum_{}^{}{k_i}=\; 0

2. \sum_{}^{}{k_ix_i}=\; \sum_{}^{}{k_iX_i}\; =\; 1

3. \sum_{}^{}{k_i^{2}}=\; \frac{1}{\sum_{}^{}{x_i^{2}}}

Assumptions of u_i;

1. E\left( u_i \right)\; =\; 0

2. Var\left( u_i \right)\; = E\left( u_i^{2} \right)\; =\; \sigma _{u}^{2}\; =\;Constant

3. u_i\; \sim \; N\left( 0, \;\sigma _{u}^{2} \right)

4. E\left( u_i,u_j \right)\; =\; 0

5. E\left( u_i,\; X_i \right)\; =\; X_i E\left(u_i \right)\; =\; 0

Properties of BLUE (Best Linear Unbiased Estimators)

Lets take the regression line Y\; =\; \alpha \; +\; \beta X_{i}\; +\; u_{i}

1. Linearity i.e., \hat{\beta}

Lets take the regression line Y\; =\; \alpha \; +\; \beta X_i\; +\; u_{i}

As we know,

\hat{\beta} = \frac{\sum_{}^{}{x_{i}y_{i}}}{\sum_{}^{}{x_{i^{}}^{2}}},

=\frac{\sum_{}^{}{\left( X- \bar{X} \right)\left( Y-\bar{Y} \right)}}{\sum_{}^{}{x_{i}^{2}}},

= \frac{\sum_{}^{}{\left( X- \bar{X} \right)Y\; -\; \sum_{}^{}{\left( X-\bar{X} \right)\bar{Y}}}}{\sum_{}^{}{x_{i}^{2}}},

=\; \frac{\sum_{}^{}{x_i\; Yi}}{\sum_{}^{}{x_{i}^{2}}},

=\; \sum_{}^{}{k_{i}Y_{i}}         because     [\frac{x_{i}}{\sum_{}^{}{x_{i}^{2}}}=\; k_{i}]

2. Unbiasedness i.e., \hat\beta  is unbiased,

From Linearity,

\hat{\beta}=\; \sum_{}^{}{k_{i}Y_{i}},

=\; \sum_{}^{}{k_{i}\; \left(\alpha \; +\; \beta X_{i}\; +\; u_{i} \right)},

=\; \alpha \sum_{}^{}{k_{i}}+\; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}},

=\; \; \beta \sum_{}^{}{k_{i}X_{i}\; +\; \sum_{}^{}{k_{i}u_{i}}}, because [\sum_{}^{}{k_i}=\; 0],

Taking expectations both sides,

E\left( \hat{\beta} \right)\; =\; \beta ,         because        [\sum_{}^{}{k_iX_i}=\; 1 \;,\; E\left( u_i \right)\; =\; 0 ],

3. Best i.e, \hat\beta is best or \hat\beta have minimum variance

As we know,

Var\left(\hat \beta \right)\; =\; E\left\{ \hat \beta \; -\beta \right\}^{2},

=\;E\; \left\{ \beta \; +\; \sum_{}^{}{k_{i}u_{i}\; +\; \beta } \right\}^{2},

=\;E\; \left\{ \sum_{}^{}{k_{i}u_{i}} \right\}^{2},

=\;E\left{k_{1}^{2}u_{1}^{2}+k_{2}^{2}u_{2}^{2}+k_{3}^{2}u_{3}^{2}........... \right\} + 2E\left\{ k_{1}k_{2}u_{1}u_{2}+ k_{2}k_{3}u_{2}u_{3} +........ \right\},

= E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\}\; +\; 2E\left\{ k_{i}k_{j}u_{i}u_{j} \right\},

=\; E\left\{ \sum_{}^{}{k_{i}^{2}u_{i}^{2}} \right\},

=\; \sum_{}^{}{k_{i}^{2}\; E\left( u_{i}^{2} \right)}, beacause       [E\left( k_{i}k_{j}u_{i}u_{j} \right)\; =\; 0],

= \sigma _{ui}^{2} \;\sum_{}^{}{k_{i}^{2}\; },

Now suppose there is another estimator,

var\left( \beta^\ast \right)\; =\; \sum_{}^{}{u_{i}^{2}}\sigma _{ui}^{2},

Where k_{i}\; \neq \; w_{i},

Let w_{i}\; =\; k_{i}\; +\; c_{i},

So, Var\; \left( \beta^\ast \right)\; =\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)^{2}\; \sigma _{ui}^{2}},

=\; \sigma _{ui}^{2}\; \sum_{}^{}{\left\{ k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right\}},

=\; \sigma _{ui}^{2}\; \sum_{}^{}{k_{i}^{2}}+\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}},  As [\sum k_{i}c_{i} = 0],

=\; Var\left( \hat{\beta} \right)\; +\; \sigma _{ui}^{2}\; \sum_{}^{}{c_{i}^{2}},

So, Var\; \left( \hat\beta \right)\; < \; Var\left( \beta^\ast \right)

Now for \alpha ;

1. Linearity i.e.; \alpha is linear,

As we know,

\hat\alpha \; =\; \bar{Y}\; -\; \hat\beta \bar{X},

=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\hat\beta \bar{X},

=\; \frac{\sum_{}^{}{Y_{i}}}{n}-\; \sum_{}^{}{k_{i}Y_{i}\bar{X}},

=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i,

So, \hat\alpha is linear.

2. Unbiasedness i.e.; \hat\alpha  is unbiased,

From linearity,

\hat\alpha=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\bar{X} \right\}} Y_i,

=\; \sum_{}^{}{\left\{ \frac{1}{n}\; -\; k_{i}\; \bar{X} \right\}\; \left( \alpha \; +\; \beta X_{i}\; +\; u_{i} \right)},

=\; \sum_{}^{}{\left\{ \frac{\alpha }{n}\; -\; \alpha k_{i}\bar{X}\; +\; \frac{\beta X_{i}}{n}\; -\; \beta X_{i}k_{i}\bar{X}\; +\; \frac{u_{i}}{n}\; -\; u_{i}k_{i}\bar{X} \right\}},

=\; \frac{n\alpha }{n }\; -\; \alpha \bar{X}\sum_{}^{}{k_{i}\; +\; \beta \; \sum_{}^{}{\frac{X_{i}}{n}\; -\; \beta \bar{X}\sum_{}^{}{k_{i}X_{i}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}-\sum_{}^{}{k_{i}u_{i}\bar{X}}}}}\; ,

=\; \alpha \; +\; \beta \bar{X}\; -\; \beta \bar{X}\; +\; \frac{\sum_{}^{}{u_{i}}}{n}\; -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}},

Now taking expectations both sides,

E\left( \hat\alpha \right)\; =\; \alpha \; +\; E\left( ui \right)\; \frac{1}{n},

E\left( \hat\alpha \right)\; =\; \alpha \; ,

3. Best i.e.; \hat\alpha is best or \hat\alpha have minimum variance,

From unbiased,

\hat\alpha =\; \alpha \; +\;\; \frac{\sum{u_i}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}},

We know,

var\left( \alpha \right)\; =\; E\left\{ \hat\alpha -\alpha \right\}^{2},

=\; E\left\{ \alpha +\frac{\sum_{}^{}{u_{i}}}{n} -\; \bar{X}\; \sum_{}^{}{k_{i}u_{i}}-\alpha \right\}^2,

=\; E\left\{ \sum_{}^{}{\left( \frac{1}{n}\; -\; k_{i}\bar{X} \right)^{2}}u_{i}^{2} \right\},

=\; \sigma _{ui}^{2}\left\{ \sum_{}^{}{\left( \frac{1}{n^{2}}\; +\; k_{i}^{2}\bar{X}\; -2\; \frac{1}{n}ki\bar{X} \right)^{}} \right\},

=\; \sigma _{ui}^{2}\left\{ \frac{n}{n^{2}}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}}-\; \frac{2}{n}\bar{X}\sum_{}^{}{ki} \right\},

=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{k_{i}^{2}} \right\},     As [\sum_{}^{}{k_i}=\; 0 ],

Lets take another estimator \alpha^\ast,

Var\left( \alpha^\ast \right)\; =\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\sum_{}^{}{w_{i}^{2}} \right\},

Where k_{i}\; \neq \; w_{i} and let w_i=k_i\;+\;c_i,

Now,

Var\left( \beta \right)\; =\; \sigma _{ui}^{2}\; \left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}\; +\; c_{i} \right)} \right\},

=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{\left( k_{i}^{2}\; +\; c_{i}^{2}\; +\; 2k_{i}c_{i} \right)} \right\},

=\; \sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{k_{i}^{2}\; +\; \bar{X}\; \sum_{}^{}{c_{i}^{2}}} \right\},           As [\sum k_{i}c_{i} = 0],

=\;\sigma _{ui}^{2}\; \frac{1}{n}+\; \sigma _{ui}^{2}\bar{X}\sum_{}^{}{k_{i}^{2}\; +\; \sigma _{ui}^{2}\bar{X}\sum_{}^{}{c_{i}^{2}}},

=\;\sigma _{ui}^{2}\left\{ \frac{1}{n}\; +\; \bar{X}\; \sum_{}^{}{c_{i}^{2}} \right\}\; +\; \sigma _{ui}^{2}\; \bar{X}\sum_{}^{}{c_{i}^{2}},

So, var\left( \hat\alpha \right)\; < \; var \left( \alpha^\ast \right)

Leave a Reply