2.4 — Goodness of Fit and Bias — Appendix

Deriving the OLS Estimators

The population linear regression model is:

$$Y_i={\beta }_0+{\beta }_1{X}_i+u_i$$

The errors $(u_i)$ are unobserved, but for candidate values of $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$, we can obtain an estimate of the residual. Algebraically, the error is:

$$\widehat{u_i}=Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i$$

Recall our goal is to find $\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ that minimizes the sum of squared errors (SSE):

$$SSE=\sum _{i=1}^n{\widehat{u_i}}^2$$

So our minimization problem is:

$$\underset{\widehat{{\beta }_0},\widehat{{\beta }_1}}{min}\sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i{)}^2$$

Using calculus, we take the partial derivatives and set it equal to 0 to find a minimum. The first order conditions are:

$$\begin{array}{rl}\frac{\partial SSE}{\partial \widehat{{\beta }_0}}& =-2{\displaystyle \sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)=0}\\ \frac{\partial SSE}{\partial \widehat{{\beta }_1}}& =-2{\displaystyle \sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)X_i=0}\end{array}$$

Finding $\widehat{{\beta }_0}$

Working with the first FOC, divide both sides by $-2$:

$${\displaystyle \sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)=0}$$

Then expand the summation across all terms and divide by $n$:

$$\underset{\overline{Y}}{\underbrace{\frac{1}{n}\sum _{i=1}^n{Y}_i}}-\underset{\widehat{{\beta }_0}}{\underbrace{\frac{1}{n}\sum _{i=1}^n\widehat{{\beta }_0}}}-\underset{\widehat{{\beta }_1}\overline{X}}{\underbrace{\frac{1}{n}\sum _{i=1}^n\widehat{{\beta }_1}{X}_i}}=0$$

Note the first term is $\overline{Y}$, the second is $\widehat{{\beta }_0}$, the third is $\widehat{{\beta }_1}\overline{X}$.¹

So we can rewrite as:

$$\overline{Y}-\widehat{{\beta }_0}-{\beta }_1=0$$

Rearranging:

$$\widehat{{\beta }_0}=\overline{Y}-\overline{X}{\beta }_1$$

Finding $\widehat{{\beta }_1}$

To find $\widehat{{\beta }_1}$, take the second FOC and divide by $-2$:

$${\displaystyle \sum _{i=1}^n(Y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{X}_i)X_i=0}$$

From the formula for $\widehat{{\beta }_0}$, substitute in for $\widehat{{\beta }_0}$:

$${\displaystyle \sum _{i=1}^n(Y_i-[\overline{Y}-\widehat{{\beta }_1}\overline{X}]-\widehat{{\beta }_1}{X}_i)X_i=0}$$

Combining similar terms:

$${\displaystyle \sum _{i=1}^n([Y_i-\overline{Y}]-[X_i-\overline{X}]\widehat{{\beta }_1})X_i=0}$$

Distribute $X_i$ and expand terms into the subtraction of two sums (and pull out $\widehat{{\beta }_1}$ as a constant in the second sum:

$${\displaystyle \sum _{i=1}^n[Y_i-\overline{Y}]X_i-\widehat{{\beta }_1}{\displaystyle \sum _{i=1}^n[X_i-\overline{X}]X_i=0}}$$

Move the second term to the righthand side:

$${\displaystyle \sum _{i=1}^n[Y_i-\overline{Y}]X_i=\widehat{{\beta }_1}{\displaystyle \sum _{i=1}^n[X_i-\overline{X}]X_i}}$$

Divide to keep just $\widehat{{\beta }_1}$ on the right:

$$\frac{{\displaystyle \sum _{i=1}^n[Y_i-\overline{Y}]X_i}}{{\displaystyle \sum _{i=1}^n[X_i-\overline{X}]X_i}}=\widehat{{\beta }_1}$$

Note that from the rules about summation operators:

$${\displaystyle \sum _{i=1}^n[Y_i-\overline{Y}]X_i={\displaystyle \sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})}}$$

and:

$${\displaystyle \sum _{i=1}^n[X_i-\overline{X}]X_i={\displaystyle \sum _{i=1}^n(X_i-\overline{X})(X_i-\overline{X})={\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}}$$

Plug in these two facts:

$$\frac{{\displaystyle \sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}=\widehat{{\beta }_1}$$

Algebraic Properties of OLS Estimators

The OLS residuals $\hat{u}$ and predicted values $\hat{Y}$ are chosen by the minimization problem to satisfy:

1. The expected value (average) error is 0:

$$E(u_i)=\frac{1}{n}{\displaystyle \sum _{i=1}^n\widehat{u_i}=0}$$

2. The covariance between $X$ and the errors is 0:

$${\hat{\sigma }}_{X,u}=0$$

Note the first two properties imply strict exogeneity. That is, this is only a valid model if $X$ and $u$ are not correlated.

3. The expected predicted value of $Y$ is equal to the expected value of $Y$:

$$\overline{\hat{Y}}=\frac{1}{n}{\displaystyle \sum _{i=1}^n\widehat{Y_i}=\overline{Y}}$$

4. Total sum of squares is equal to the explained sum of squares plus sum of squared errors:

$$\begin{array}{rl}TSS& =ESS+SSE\\ \sum _{i=1}^n(Y_i-\overline{Y}{)}^2& =\sum _{i=1}^n(\widehat{Y_i}-\overline{Y}{)}^2+\sum _{i=1}^n{u}^2\end{array}$$

Recall $R^2$ is $\frac{ESS}{TSS}$ or $1-SSE$

5. The regression line passes through the point $(\overline{X},\overline{Y})$, i.e. the mean of $X$ and the mean of $Y$.

Bias in $\widehat{{\beta }_1}$

Begin with the formula we derived for $\widehat{{\beta }_1}$:

$$\widehat{{\beta }_1}=\frac{{\displaystyle \sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}$$

Recall from Rule 6 of summations, we can rewrite the numerator as

$$\begin{array}{rl}=& {\displaystyle \sum _{i=1}^n(Y_i-\overline{Y})(X_i-\overline{X})}\\ =& {\displaystyle \sum _{i=1}^n{Y}_i(X_i-\overline{X})}\end{array}$$

$$\widehat{{\beta }_1}=\frac{{\displaystyle \sum _{i=1}^n{Y}_i(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}$$

We know the true population relationship is expressed as:

$$Y_i={\beta }_0+{\beta }_1{X}_i+u_i$$

Substituting this in for $Y_i$ in equation 2:

$$\widehat{{\beta }_1}=\frac{{\displaystyle \sum _{i=1}^n({\beta }_0+{\beta }_1{X}_i+u_i)(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}$$

Breaking apart the sums in the numerator:

$$\widehat{{\beta }_1}=\frac{{\displaystyle \sum _{i=1}^n{\beta }_0(X_i-\overline{X})+{\displaystyle \sum _{i=1}^n{\beta }_1{X}_i(X_i-\overline{X})+{\displaystyle \sum _{i=1}^n{u}_i(X_i-\overline{X})}}}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}$$

We can simplify equation 4 using Rules 4 and 5 of summations

1. The first term in the numerator $\left[{\displaystyle \sum _{i=1}^n{\beta }_0(X_i-\overline{X})}\right]$ has the constant ${\beta }_0$, which can be pulled out of the summation. This gives us the summation of deviations, which add up to 0 as per Rule 4:

$$\begin{array}{rl}{\displaystyle \sum _{i=1}^n{\beta }_0(X_i-\overline{X})}& ={\beta }_0{\displaystyle \sum _{i=1}^n(X_i-\overline{X})}\\ & ={\beta }_0(0)\\ & =0\end{array}$$

2. The second term in the numerator $\left[{\displaystyle \sum _{i=1}^n{\beta }_1{X}_i(X_i-\overline{X})}\right]$ has the constant ${\beta }_1$, which can be pulled out of the summation. Additionally, Rule 5 tells us ${\displaystyle \sum _{i=1}^n{X}_i(X_i-\overline{X})={\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}$:

$$\begin{array}{rl}{\displaystyle \sum _{i=1}^n{\beta }_1{X}_1(X_i-\overline{X})}& ={\beta }_1{\displaystyle \sum _{i=1}^n{X}_i(X_i-\overline{X})}\\ & ={\beta }_1{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}\end{array}$$

When placed back in the context of being the numerator of a fraction, we can see this term simplifies to just ${\beta }_1$:

$$\begin{array}{rl}\frac{{\beta }_1{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}& =\frac{{\beta }_1}{1}\times \frac{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}\\ & ={\beta }_1\end{array}$$

Thus, we are left with:

$$\widehat{{\beta }_1}={\beta }_1+\frac{{\displaystyle \sum _{i=1}^n{u}_i(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}$$

Now, take the expectation of both sides:

$$E[\widehat{{\beta }_1}]=E[{\beta }_1+\frac{{\displaystyle \sum _{i=1}^n{u}_i(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}]$$

We can break this up, using properties of expectations. First, recall $E[a+b]=E[a]+E[b]$, so we can break apart the two terms.

$$E[\widehat{{\beta }_1}]=E[{\beta }_1]+E\left[\frac{{\displaystyle \sum _{i=1}^n{u}_i(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}\right]$$

Second, the true population value of ${\beta }_1$ is a constant, so $E[{\beta }_1]={\beta }_1$.

Third, since we assume $X$ is also “fixed” and not random, the variance of $X$, ${\displaystyle \sum _{i=1}^n(X_i-\overline{X})}$, in the denominator, is just a constant, and can be brought outside the expectation.

$$E[\widehat{{\beta }_1}]={\beta }_1+\frac{E\left[{\displaystyle \sum _{i=1}^n{u}_i(X_i-\overline{X})}\right]}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}$$

Thus, the properties of the equation are primarily driven by the expectation $E[{\displaystyle \sum _{i=1}^n{u}_i(X_i-\overline{X})]}$. We now turn to this term.

Use the property of summation operators to expand the numerator term:

$$\begin{array}{rl}\widehat{{\beta }_1}& ={\beta }_1+\frac{{\displaystyle \sum _{i=1}^n{u}_i(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}\\ \widehat{{\beta }_1}& ={\beta }_1+\frac{{\displaystyle \sum _{i=1}^n(u_i-\overline{u})(X_i-\overline{X})}}{{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}\end{array}$$

Now divide the numerator and denominator of the second term by $\frac{1}{n}$. Realize this gives us the covariance between $X$ and $u$ in the numerator and variance of $X$ in the denominator, based on their respective definitions.

$$\begin{array}{rl}\widehat{{\beta }_1}& ={\beta }_1+\frac{\frac{1}{n}{\displaystyle \sum _{i=1}^n(u_i-\overline{u})(X_i-\overline{X})}}{\frac{1}{n}{\displaystyle \sum _{i=1}^n(X_i-\overline{X}{)}^2}}\\ \widehat{{\beta }_1}& ={\beta }_1+\frac{cov(X,u)}{var(X)}\\ \widehat{{\beta }_1}& ={\beta }_1+\frac{s_{X,u}}{s_X^2}\end{array}$$

By the Zero Conditional Mean assumption of OLS, $s_{X,u}=0$.

Alternatively, we can express the bias in terms of correlation instead of covariance:

$$E[\widehat{{\beta }_1}]={\beta }_1+\frac{cov(X,u)}{var(X)}$$

From the definition of correlation:

$$\begin{array}{rl}cor(X,u)& =\frac{cov(X,u)}{s_X{s}_u}\\ cor(X,u)s_X{s}_u& =cov(X,u)\end{array}$$

Plugging this in:

$$\begin{array}{rl}E[\widehat{{\beta }_1}]& ={\beta }_1+\frac{cov(X,u)}{var(X)}\\ E[\widehat{{\beta }_1}]& ={\beta }_1+\frac{[cor(X,u)s_x{s}_u]}{s_X^2}\\ E[\widehat{{\beta }_1}]& ={\beta }_1+\frac{cor(X,u)s_u}{s_X}\\ E[\widehat{{\beta }_1}]& ={\beta }_1+cor(X,u)\frac{s_u}{s_X}\end{array}$$

Proof of the Unbiasedness of $\widehat{{\beta }_1}$

Begin with equation:²

$$\widehat{{\beta }_1}=\frac{\sum Y_i{X}_i}{\sum X_i^2}$$

Substitute for $Y_i$:

$$\widehat{{\beta }_1}=\frac{\sum ({\beta }_1{X}_i+u_i)X_i}{\sum X_i^2}$$

Distribute $X_i$ in the numerator:

$$\widehat{{\beta }_1}=\frac{\sum {\beta }_1{X}_i^2+u_i{X}_i}{\sum X_i^2}$$

Separate the sum into additive pieces:

$$\widehat{{\beta }_1}=\frac{\sum {\beta }_1{X}_i^2}{\sum X_i^2}+\frac{u_i{X}_i}{\sum X_i^2}$$

${\beta }_1$ is constant, so we can pull it out of the first sum:

$$\widehat{{\beta }_1}={\beta }_1\frac{\sum X_i^2}{\sum X_i^2}+\frac{u_i{X}_i}{\sum X_i^2}$$

Simplifying the first term, we are left with:

$$\widehat{{\beta }_1}={\beta }_1+\frac{u_i{X}_i}{\sum X_i^2}$$

Now if we take expectations of both sides:

$$E[\widehat{{\beta }_1}]=E[{\beta }_1]+E\left[\frac{u_i{X}_i}{\sum X_i^2}\right]$$

${\beta }_1$ is a constant, so the expectation of ${\beta }_1$ is itself.

$$E[\widehat{{\beta }_1}]={\beta }_1+E[\frac{u_i{X}_i}{\sum X_i^2}]$$

Using the properties of expectations, we can pull out $\frac{1}{\sum X_i^2}$ as a constant:

$$E[\widehat{{\beta }_1}]={\beta }_1+\frac{1}{\sum X_i^2}E[\sum u_i{X}_i]$$

Again using the properties of expectations, we can put the expectation inside the summation operator (the expectation of a sum is the sum of expectations):

$$E[\widehat{{\beta }_1}]={\beta }_1+\frac{1}{\sum X_i^2}\sum E[u_i{X}_i]$$

Under the exogeneity condition, the correlation between $X_i$ and $u_i$ is 0.

$$E[\widehat{{\beta }_1}]={\beta }_1$$

Footnotes

1. From the rules about summation operators, we define the mean of a random variable $X$ as $\overline{X}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{X}_i}$. The mean of a constant, like ${\beta }_0$ or ${\beta }_1$ is itself.

2. Admittedly, this is a simplified version where $\widehat{{\beta }_0}=0$, but there is no loss of generality in the results.