4.1 Stochastic regressors
In most economic applications, the A1 of the fixed regressors cannot be retained in the data generating process. For instance, the DGP of a particular country in the basic Keynesian model is explained by the final consumption and the investment as regressors. These regressors may be stochastic variables as they depend on other known and unknown random factors. Therefore, how would the absence of this assumption affect the different estimators and statistic tests?
Let us consider our model (3.2) again:
\[\begin{equation*} y=X\beta+\varepsilon \end{equation*}\]
Now, even the matrix \(X\) is stochastic, the estimators still computed as before:
\[\begin{equation*} \widehat\beta=\big(X^tX)^{-1}X^ty \end{equation*}\]
Whereas the computation of the means and the variances will not progress and remain at the following step (see the subsections 3.2.2 and 3.2.3):
\[\begin{equation} E(\widehat\beta)=\beta+E\bigg[\big(X^tX\big)^{-1}X^t\varepsilon\bigg] \tag{4.1} \end{equation}\] \[\begin{equation} Var(\widehat\beta)=E\bigg[\big(X^tX\big)^{-1}X^t\varepsilon\varepsilon^tX\big(X^tX\big)^{-1}\bigg] \tag{4.2} \end{equation}\]
The above formulas show that unbiasedness and efficiency are not guaranteed since the matrix \(X\) is stochastic, and hence the quantity \(\big(X^tX\big)^{-1}X^t\) cannot be removed outside the expectation operator. Therefore, the assumption A2 (\(E(\varepsilon)=0\)) cannot be used (unless \(X\) and \(\varepsilon\) are independent represented by the assumption A5) to enable the unbiasedness of the estimator vector \(\widehat\beta\).
The same holds for the matrix variance as A3 and A4 cannot be used, and hence \(\widehat\beta\) will no longer be efficient.
According to what was said in the previous chapter that the A5: \(E(X^t\varepsilon)=0\) is not needed if A1 of fixed regressor is satisfied, but with random regressors, that assumption must be retained to get the unbiasedness.
However, Probability theory provides an alternative way to save the same results under A1 and A5 based on the conditionality. That is deriving the estimators and the disturbances conditional on \(X\) so that those estimators still unbiased and efficient. Thus, the conditional mean is given by:
\[\begin{align} E(\widehat\beta/X)&=\beta+E\bigg[(X^tX)^{-1}X^t\varepsilon/X\bigg]\notag\\ &=\beta+(X^tX)^{-1}X^t\overbrace{E(\varepsilon/X)}^{=0}=\beta \tag{4.3} \end{align}\]
And the conditional variance is given by:
\[\begin{align} Var(\widehat\beta/X)&=E\bigg[(X^tX)^{-1}X^t\varepsilon\varepsilon^tXX(X^tX)^{-1}/X\bigg]\notag\\ &=(X^tX)^{-1}X^t\overbrace{E(\varepsilon\varepsilon^t/X)}^{=\sigma^2}X(X^tX)^{-1}=\sigma^2(X^tX)^{-1} \tag{4.4} \end{align}\]
Unfortunately, this alternative cannot be considered as a solution as it will be hard to get reliable inferences from a specific sample selected randomly because all the obtained results will be related only to that particular sample.
::: {.definition #conditional-expectation name=“properties of the conditional expectation operator”}
Let \(X\) and \(Y\) be two random variables, and \(\Phi(X,Y)\) be an arbitrary function of \(X\) and \(Y\):
- \(E\bigg(\Phi(X,Y)\bigg)=\underset{x}{E}\bigg[\underset{y}{E}\big(\Phi(X,Y)/X\big)\bigg]\).
- \(Var\bigg(\Phi(X,Y)\bigg)=\underset{x}{E}\bigg[\underset{y}{Var}\big(\Phi(X,Y)/X\big)\bigg]+\underset{x}{Var}\bigg[\underset{y}{E}\big(\Phi(X,Y)/X\big)\bigg]\).
Wher \(\underset{x}{E}\) is the expectation with respect to \(X\), and \(\underset{Y}{E}\) with respect to \(Y\).
:::
4.1.1 Statistical properties of the estimators
Since \(X\) and \(\varepsilon\) are now random variables, they can either be dependently or independently distributed. In the former case, the estimators cannot be unbiased as \(E\big((X^tX)^{-1}X^t\varepsilon\big)\not=0\). While in the latter, the expressions (4.1) and (4.2), after using A2, A3, A4, and the definition ??, will be rewritten as follows:
\[\begin{align*} E(\widehat\beta)&=\beta+\underset{x}{E}\bigg[\underset{\varepsilon}{E}\bigg((X^tX)^{-1}X^t\varepsilon/X\bigg)\bigg]\\ &=\beta+\underset{x}{E}\bigg[(X^tX)^{-1}X^t\overbrace{\underset{\varepsilon}{E}(\varepsilon/X)}^{=0}\bigg]\\ &=\beta \end{align*}\]
\[\begin{align*} Var(\widehat\beta)&=\underset{x}{E}\bigg[Var\big(\widehat\beta/X\big)\bigg]+\underset{x}{Var}\bigg[E\big(\widehat\beta/X\big)\bigg]\\ &=\underset{x}{E}\bigg[\sigma^2\big(X^tX\big)^{-1}\bigg]+\overbrace{\underset{x}{Var}\big(\beta\big)}^{=0}\\ &=\sigma^2E\bigg[\big(X^tX\big)^{-1}\bigg] \end{align*}\]
The first formula shows that if the regressors and the error term are independently distributed, then the OLS estimators still unbiased. However, as the second formula shows, the variance matrix, and hence the efficiency of those estimators, depends on the distribution of \(X\). Therefore, in order to maintain the efficiency, the matrix \(X\) should not vary drastically to allow the stability of the expected value \((X^tX)^{-1}\).
In many cases, it is impossible to determine precisely the joint distribution of the regressors in \(X\), which makes hard the calculation of the estimates along with their expectations and variances. Moreover, the required statistic tests will no longer follow the classical distributions such as t-student and Fisher. Therefore, more assumptions about the matrix \(X\) should be made to get reliable estimates.
Definition 4.1 Consistency: Let \(\widehat\beta_n\), computed from a sample with \(n\) observations, be the estimator of the population parameter \(\beta\), we say that \(\beta\) is the probability limit of \(\widehat\beta_n\) if:
For all \(c>0 : \underset{n\to +\infty}{lim} p\bigg[(\widehat\beta_n-\beta)<c\bigg]=1\), simplified by: \(plim(\widehat\beta_n)=\beta\).
Then \(\widehat\beta_n\) will be called consistent estimator.
Properties of \(plim\):
- \(plim(A+B)=plim(A)+plim(B)\).
- \(plim(AB)=plim(A)\times plim(B)\).
- \(plim(\frac{A}{B})=\frac{plim(A)}{plim(B)},\,\,\,plim(B)\not=0\).
- \(plim\,\, \mathrm g(A)=\mathrm g(plim(A))\), \(\mathrm g\) an arbitrary function.
Low of large numbers: Let \(X_t\to IID\) identically and independently distributed, \(E(X_t)=\mu\) the population mean, \(\mu_r=E\bigg[(X_t-\mu)^r\bigg]\) the population centered moment (of order \(r\)), \(\overline X_n=\frac{1}{n}\sum\limits^n_{t=1}X_t\) the sample mean, \(\frac{1}{n}\sum\limits^n_{t=1}\big(X_t-\overline X_n\big)^r\) the sample centered moment (of order \(r\)). Using the low of large numbers, we have:
- \(plim\big(\overline X_n\big)=plim\bigg(\frac{1}{n}\sum\limits^n_{t=1}X_t\bigg)=\mu\).
- \(plim\bigg(\frac{1}{n}\sum\limits^n_{t=1}\big(X_t-\overline X_n\big)^r\bigg)=\mu_r\).
Central limit theorem: if \(\overline X_n\) have an arbitrary distribution, then the following transformed random variable:
\(\sqrt{n}\frac{\big(\overline X_n-\mu\big)}{\sigma}\overset{d}{\to}\mathrm N\big(0,1\big)\).
is asymptotically normally distributed.
4.1.2 Asumptotic analysis and consistency
We know from the low of large numbers and probability limit that any sample moment converges to its corresponding population moment. Formally speaking, They are equal asymptotically. The central limit theorem also states that any arbitrary distribution can, asymptotically, equals the normal distribution after a small transformation in the corresponding random variable. Consequently, we can use the asymptotic analysis to check the statistical property of our OLS estimators and perform the required statistic tests.
4.1.2.1 Consistency
Let us start where we left off:
\[\begin{equation*} \widehat\beta=\beta+\big(X^tX\big)^{-1}X^t\varepsilon \end{equation*}\]
With a slight transformation the above formula will be:
\[\begin{equation} \widehat\beta=\beta+\bigg(\frac{1}{n}X^tX\bigg)^{-1}\frac{1}{n}X^t\varepsilon \tag{4.5} \end{equation}\]
Consider \(\frac{1}{n}X^tX\) and \(\frac{1}{n}X^t\varepsilon\) as a moments, and suppose that the sample size is infinite (\(n\to +\infty\)). Unbiasedness, thus, requires that \(\frac{1}{n}X^tX\) must be a fixed quantity:
\[\begin{equation*} \frac{1}{n}X^tX=Q \end{equation*}\]
Where \(Q\) is \((k\times k)\) matrix invertible fixed matrix. This result called stability condition. Whereas, the second moment \(\frac{1}{n}X^t\varepsilon\) must be equal to zero:
\[\begin{equation*} \frac{1}{n}X^t\varepsilon=0 \end{equation*}\]
Which is called orthogonality condition.
In practice, there is no infinite sample size, but with a sample size large enough, we can make use of the probability limit to get the same above results by approximation as follows:
\[\begin{equation} plim\bigg(\frac{1}{n}X^tX\bigg)=Q \tag{4.6} \end{equation}\]
\[\begin{equation} plim\bigg(\frac{1}{n}X^t\varepsilon\bigg)=0 \tag{4.7} \end{equation}\]
The first result will be adopted as an assumption that replaces A1. It will be denoted A1.1. The second one is another interpretation of A5 stated in the previous chapter. Finally, we get the consistency (not the unbiasedness) as follows:
\[\begin{align*} plim\big(\widehat\beta\big)&=\beta+plim\Bigg[\bigg(\frac{1}{n}X^tX\bigg)^{-1}\frac{1}{n}X^t\varepsilon\Bigg]\\ &=\beta+plim\bigg(\frac{1}{n}X^tX\bigg)^{-1}plim\bigg(\frac{1}{n}X^t\varepsilon\bigg)\\ &=\beta+\Bigg(\overbrace{plim\bigg(\frac{1}{n}X^tX\bigg)}^{=Q}\Bigg)^{-1}plim\bigg(\frac{1}{n}X^t\varepsilon\bigg)\\ &=\beta+Q^{-1}\overbrace{plim\bigg(\frac{1}{n}X^t\varepsilon\bigg)}^{=0}\\ &=\beta \end{align*}\]
Once these results have been verified, we can say that the OLS estimators are consistent.
There is no way to perform a test used to detect the orthogonality condition (dependence between \(X\) and \(\varepsilon\)) by using the residuals vector \(e\) because the OLS criterion implies that orthogonality in the sample by the normal equation \(X^te=0\)
4.1.2.2 Efficiency
Using the same above results, (4.6) and (4.7), the probability limit of the estimated variance \(s^2\) will be:
\[\begin{align} plim \big(s^2\big)&=plim\bigg(\frac{e^te}{n-k}\bigg)\notag\\ &=plim\bigg(\frac{\varepsilon^t M\varepsilon}{n-k}\bigg)\notag\\ &=plim\Bigg(\frac{\varepsilon^t\big(I-X(X^tX)^{-1}X^t\big)\varepsilon}{n-k}\Bigg)\notag\\ &=plim\Bigg(\frac{\varepsilon^t\varepsilon-\varepsilon^tX(X^tX)^{-1}X^t\varepsilon}{n-k}\Bigg)\notag\\ &=\overbrace{plim\bigg(\frac{n}{n-k}\bigg)}^{=1}\Bigg(\overbrace{plim\bigg(\frac{1}{n}\varepsilon^t\varepsilon\bigg)}^{=\sigma^2I}-\overbrace{plim\bigg(\frac{1}{n}\varepsilon^tX\bigg)}^{=0}\overbrace{plim\bigg(\frac{1}{n}X^tX\bigg)^{-1}}^{=Q^{-1}}\overbrace{plim\bigg(\frac{1}{n}X^t\varepsilon\bigg)}^{=0}\Bigg)\notag\\ &=\sigma^2 \tag{4.8} \end{align}\]
Which means that the estimated variance \(s^2\) is also a consistent estimator.
Asymptotically, there is no difference between \(s^2=\frac{e^te}{n-k}\) and \(\widehat \sigma^2=\frac{e^te}{n}\) since both are consistent:
\(plim\bigg(\frac{e^te}{n-k}\bigg)=plim\bigg(\frac{e^te}{n}\bigg)\)
4.1.2.3 Distribution of \(\widehat\beta\)
Under the assumption of stochastic regressors and the orthogonality condition, the distribution of \(\widehat\beta\) depends on both, the distribution of \(X\) and that of \(\varepsilon\). If we suppose that both \(X\) and \(\varepsilon\) are identically and independently distributed, generally with an arbitrary distribution, we can then use the central limit theorem to get asymptotic normal distribution as follows:
\[\begin{equation*} \sqrt{n}\big(\widehat\beta-\beta\big)=\frac{1}{\sqrt{n}}\bigg(\frac{1}{n}X^tX\bigg)^{-1}\big(X^t\varepsilon\big) \end{equation*}\]
As \(\widehat\beta\) is consistent, the mean of that variable is \(plim\bigg(\sqrt{n}\big(\widehat\beta-\beta\big)\bigg)=0\), and using the fact that \(Q^t=Q\), its variance will be:
\[\begin{align*} Var\bigg(\sqrt{n}\big(\widehat\beta-\beta\big)\bigg)&\to \frac{1}{\sqrt{n}}Q^{-1}X^t\sigma^2\frac{1}{\sqrt{n}}XQ^{-1}=\sigma^2Q^{-1}\overbrace{QQ^{-1}}^{=I}\\ Var\bigg(\sqrt{n}\big(\widehat\beta-\beta\big)\bigg)&\to \sigma^2Q^{-1} \end{align*}\]
Therefore the asymptotic distribution of the transformed variable \(\sqrt{n}\big(\widehat\beta-\beta\big)\) will be:
\[\begin{equation} \sqrt{n}\big(\widehat\beta-\beta\big)\overset{d}{\to}\mathrm N\big(0, \sigma^2Q^{-1}\big) \tag{4.9} \end{equation}\]
Hence, the asymptotic distribution of the original variable \(\widehat\beta\) will be:
\[\begin{equation} \widehat\beta\overset{d}{\to}\mathrm N\bigg(\beta, \frac{1}{n}\sigma^2Q^{-1}\bigg) \tag{4.10} \end{equation}\]
In conclusion, the asymptotic distribution can be used by approximation in large samples. However, the required sample size suggested by many econometricians to justify the approximation should be greater than 50 (after subtracting the number of regressors \(n-k>50\)).
4.1.2.4 Statistic tests:
As you might noticed, The distribution of \(\widehat\beta\) (4.10) cannot be used directly as the matrix \(Q\) is unknown. However, it can be approximated by \(\frac{1}{n}(X^tX)\) computed from the sample at hand. If so, the distribution of \(\widehat\beta\) will be approximated by:
\[\begin{equation} \widehat\beta\approx\mathrm N\bigg(\beta,\sigma^2(X^tX)^{-1}\bigg) \tag{4.11} \end{equation}\]
Based on this result, t-student will and F-test will save their standard distributions but by approximation. However, since t-student tends to the normal distribution in large samples, it can be then replaced by the z-score.