2.4 Hypothesis testing
As shown earlier, OLS estimator \(\widehat \beta_1\) expressed in (2.12) (\(\widehat \beta_0\) in (2.11)) is the best estimator of the population parameter \(\beta_1\) (\(\beta_0\)) under the classical assumptions 2.8. However, since different samples give different parameter estimates, the unbiasedness requires these estimates to lie around the population parameter, and the efficiency guarantees to have the highest possible certainty (or lower volatility) compared to other unbiased estimators.
Since now, we know the distribution of the difference between the true parameter value and estimated value \(\widehat\beta_0-\beta_0\) (scaled by \(s_{\widehat\beta_1}\)), we do not have to do the hard work (sometimes impossible) to collecting many samples then check the mean if closer to zero or not. Instead, we can use only the sample at hand, then using the student distribution to get the probability of the difference (2.42) such that higher values give more confidence such that the theoretical parameter is correctly estimated.
One critical issue not yet solved is how to compute that difference since the true parameter value \(\beta_1\) (or \(\beta_0\)) is unknown? It turns out that the primary purpose of the estimated model is to check some prespecified value, say \(\beta_1^0\), for the true parameter so that if the corresponding probability is high enough, then we adopt it as a possible value for the population parameter.
Setting \(\beta_1=\beta_1^0\) is called The null hypothesis denoted:
\[\begin{equation*} H_0:\beta_1=\beta_1^0 \end{equation*}\]
As a consequence we have also the alternative hypothesis when \(\beta_1\neq \beta_1^0\) denoted:
\[\begin{equation*} H_1:\beta_1\neq \beta_1^0 \end{equation*}\]
Note that the null hypothesis above written in that form is called Two sided hypothesis to distinguish from One sided hypothesis written with one of the following:
\[\begin{equation*} \beta_1 < \beta_1^0,\quad\beta_1\leqslant \beta_1^0,\quad\beta_1> \beta_1^0,\quad\beta_1\geqslant \beta_1^0 \end{equation*}\]
2.4.1 Test the significance of estimates:
The simplest form of the null hypothesis is to check only if the regressor affects or not the dependent variable. In formal term check the significance of the regressor. The two hypotheses then will be denoted:
\[\begin{equation*} H_0:\beta_1=0\\ H_1:\beta_1\neq 0 \end{equation*}\]
Due to the sampling process, it is rare when the difference \(\widehat\beta_1-\beta_1^0\) precisely equals zero, some difference often exists. The question is whether this difference is small enough to be considered as the result of the randomness process leading the nonrejection of the null hypothesis or the two parameters are indeed different so that rejecting it.
Note that, in the econometric literature, we often use the term not reject instead of accept. Because the null hypothesis is supposed to represent the current state that has been existing for a while with some familiarity, so rejecting this state against other alternatives requires having strong evidence. Statistically, the rejection of the null hypothesis requires lower probability values.
If we take the null hypothesis \(H_0:\beta_1=0\), the possible Values of \(\widehat \beta_1\) can be divided into two regions, the randomness region that is around zero (in general around \(\beta_1^0\)), and the region of significant differences that is far away from zero (positively or negatively). Thus, we compute the two-sided statistic \(|t^c|=|\frac{\widehat\beta_1}{s_{\widehat\beta_1}}|\), using the absolute values for convenience. But how can those regions be determined?
The answer depends on the confidence level (or the significance level ) chosen by the investigator, with 95% (or 5%) level (the most level used in practice), for instance, the bound that separates the regions, defined in terms for \(t^c\), can be computed using the student probability as follows:
\[\begin{align} &P\bigg(|t^c_{\widehat\beta_1}|\leqslant t^{0.05}_{n-2}\bigg)=0.95\quad in\quad general:\\ &P\bigg(|t^c_{\widehat\beta_1}|\leqslant t^{\alpha}_{n-2}\bigg)=1-\alpha \tag{2.44} \end{align}\]
Where \(t^{\alpha}_{n-2}\) is the tabulated1 value, called The critical region, that depends on the sample size and the significance level selected by the investigator, and \(\alpha\) is the significance level.
Note that the above expression can be rewritten \(P\bigg(-t_{n-2}^{0.05}\leqslant t_{\widehat\beta_1}^{c}\leqslant t_{n-2}^{0.05}\bigg)=0.95\) which yields two equally regions in absolute values, one positive and one negative.
The first simple way to decide to reject or not the null hypothesis is by comparing the computed value \(t_{\widehat\beta_1}^{c}\) to the tabulated one \(t_{n-2}^{0.05}\) (based on the confidence level 95%) so that we have two cases:
- \(\bigg|t_{\widehat\beta_1}^{c}\bigg|\leqslant t_{n-2}^{0.05}\): We do not reject the null hypothesis with 95% of confidence level (or 5% of significance level).
- \(\bigg|t_{\widehat\beta_1}^{c}\bigg|\geqslant t_{n-2}^{0.05}\) : We reject the null hypothesis.
Definition 2.12 (Type of errors)
If the DGP is inline with the null hypothesis, say \(\beta_1=0\), and the test does not reject the null hypothesis, say with a confidence level of \(0.95\) (denoted also \(1-\alpha\)), then this decision is correct. However, this also means that the test rejects the null hypothesis with the remaining \(0.05\) (\(\alpha\)) which is an incorrect decision.
The siginficance level \(\alpha\) must be defined before starting the testing process.
The incorrect decision of rejecting the null hypothesis when it is true is called Error of the first type or also The false positive.
The incorrect decision of nonrejecting the null hypothesis when it is false is called The error of the second type denoted by \(\beta\)
The correct decision of rejecting the null hypothesis when it is false is called The power of the test denoted by \(1-\beta\).
2.4.2 Confidence intervals:
Instead of testing a prespecified values of the true parameter \(\beta_1\), we define all the possible values without causing the rejection of the null hypothesis as follows:
\[\begin{align*} -t_{n-2}^{\alpha}\leqslant \frac{\widehat \beta_1-\beta_1}{s_{\widehat\beta_1}}\leqslant t_{n-2}^{\alpha}&\implies -t_{n-2}^{\alpha}.s_{\widehat\beta_1}\leqslant \widehat \beta_1-\beta_1 \leqslant t_{n-2}^{\alpha}.s_{\widehat\beta_1}\\ &\implies \widehat \beta_1-t_{n-2}^{\alpha}.s_{\widehat\beta_1}\leqslant \beta_1 \leqslant \widehat \beta_1+t_{n-2}^{\alpha}.s_{\widehat\beta_1} \end{align*}\]
That is, the possible values for the true parameter provided by the estimated model, given the significance \(\alpha\), are inside the following interval:
\[\begin{equation} \beta_1\in \bigg[\widehat \beta_1-t_{n-2}^{\alpha}.s_{\widehat\beta_1}\quad,\quad \widehat \beta_1+t_{n-2}^{\alpha}.s_{\widehat\beta_1}\bigg] \tag{2.45} \end{equation}\]
This interval typically is more convenient for testing the null hypothesis such that the validity of any hypothetical population parameter value can be readily verified whether it belongs to this interval or not. For instance, for the significance test, if zero belongs to the interval then the null hypothesis \(H_0:\beta_1=0\) will not be rejected.
Now since the estimator \(\widehat\beta_1\), the estimator standard error \(s_{\widehat\beta_1}\), and the sample size \(n\) are all known, The width of the above interval depends only on the tabulated value \(t_{n-2}^{\alpha}\) that, in turn, depends only on the significance level \(\alpha\). By inspecting the statistic table of the student, we see that the decrease of \(\alpha\) leads to an increase of \(t_{n-2}^{\alpha}\), and hence the interval becomes wider yields in less certainty. That is:
\[\begin{equation*} \left(t_{18}^{0.2}=1.330\right)<\left(t_{18}^{0.1}=1.734\right)<\left(t_{18}^{0.05}=2.101\right)<\left(t_{18}^{0.01}=2.878\right) \end{equation*}\]
All the previous analysis holds true for the intercept estimate so that its confidence interval is:
\[\begin{equation} \beta_0\in \bigg[\widehat \beta_0-t_{n-2}^{\alpha}.s_{\widehat\beta_0}\quad,\quad \widehat \beta_0+t_{n-2}^{\alpha}.s_{\widehat\beta_0}\bigg] \tag{2.46} \end{equation}\]
Instead of fixing the size \(\alpha\) beforehand, it can be left unspecified and computing the minimum probability of the computed test \(t^c_{\widehat\beta_1}\) that leads to the rejection of the null hypothesis. Formally, The student probability of \(t^c_{\widehat\beta_1}=t^{\alpha}_{n-2}\) is called p-value, so any value smaller than that leads to reject the null hypothesis.
The main problem with p-value(all 2019) is the misunderstanding and misusing of this p-value when we decide to reject the null hypothesis based on some threshold, from which we wrongly interpreting it as the probability of rejecting the null hypothesis.
By using again the example 2.1, we test the significance of the slop, the true value 4, and a wrong value, say 3.
Test of the significance \(H_0:\beta_1=0\):
We cant test the significance of the slop in three ways:
compare the computed \(t^c_{\widehat\beta_1}\) to the tabulated one \(t^{\alpha}_{n-2}\) with the (default) significance level 5%.
# extract the t-computed
t_c <- summary(model_ols)$coefficients[[2, 3]]
# compute the two sided t-tabulated with 5% level
# and n-2=148
t_t <- qt(1 - (0.05 / 2), 148)
paste('t-computed', ':', abs(t_c), 't-tabulated', ':', abs(t_t))#> [1] "t-computed : 75.5390395620822 t-tabulated : 1.97612249361374"Since \(t^c_{\widehat\beta_1}> t^{\alpha}_{n-2}\) we reject the null hypothesis so that the coefficient is significant.
Using the confidence interval:
#> 2.5 % 97.5 %
#> 3.942119 4.167672Again, since zero does not belong to this interval then we say that the coefficient is significant.
Using p-value:
#> [1] 5.584012e-24This value is a very tiny leading to the rejection of the null hypothesis, which again means the coefficient is significant.
\(H_0:\beta_1=4\):
the simplest way to check this hypothesis is by verifying whether 4 belongs to this interval or not
#> 2.5 % 97.5 %
#> 3.942119 4.167672Indeed it is inside the interval, so we can say with confidence (95%) that the true value is 4. be aware that this all that says is that if we repeat the sampling process many times, then the value 4 would exist in 95% of the resulted confidence intervals.
\(H_0:\beta_1=3\):
By the same way, if we check the value 3, it is outside this interval, which means it can not be the true value with confidence (95%)
Test of the significance \(H_0:\beta_1=0\):
We make use of the scipy package to get the t-tabulated. Compare the t-computed to the t-tabulated.
#> <string>:1: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`t_tabulated = scipy.stats.t.ppf(q=1-(0.05/2), df=df)
print('t_computed: {}\nt_tabulated: {}'.format(t_computed,t_tabulated))#> t_computed: 82.18600007568621
#> t_tabulated: 2.10092204024096Using the confidence interval
#> 0 3.819529
#> 1 4.019929
#> Name: xP, dtype: float64Using p-value
#> 1.2288085474099391e-242.4.3 test the overall significance of the model
Basically, in the case of the sample linear regression, the model contains only one explanatory variable. As the constant term is not much more interesting, testing thus the variable coefficient significance is the same as testing the overall model significance. In contrast with the general linear model with more than one regressor, the overall significance of all the regressor coefficients differs from separately testing each regressor. Because the model, simultaneously, could be significant as a whole, and some of its regressors are not significant. What concerns us usually is testing only the total explanation of regressors as a whole. Therefore we will derive a test for the significance of the model as a whole. Despite, for our case, we do not need this test, but we show that it gives the same result as the t-test.
Using (2.36), (2.40), and 2.11, The fisher statistic, denoted by F-test, will be computed as follows:
\[\begin{align} F(1,n-2)&=\frac{\Bigg(\frac{\left(\widehat\beta_1-\beta_1\right)^2}{\sigma^2\big/\sum \left(X_t-\overline X\right)}\Bigg)}{\frac{\sum e_t^2}{\sigma^2}\bigg/(n-2)}\notag\\ &=\frac{\left(\widehat\beta_1-\beta_1\right)^2\sum\left(X_t-\overline X\right)^2}{\sum e_t^2\big/(n-2)} \tag{2.47} \end{align}\]
Notice that the numerator is a \(\chi^2\) divided by 1, and the denominator is another \(\chi^2\) but divided by \(n-2\), which means that fisher, unlike t-test, has two degrees of freedom.
For the simple linear regression, it happens that the square of the t-test is equal to F-test.
\[\begin{align} t^2_{(n-2)}&=\frac{\left(\widehat\beta_1-\beta_1\right)^2\bigg/\sigma^2_{\widehat\beta_1}}{(n-2)s^2_{\widehat\beta_1}\bigg/\sigma^2_{\widehat\beta_1}(n-2)}\notag \\ &=\frac{\left(\widehat\beta_1-\beta_1\right)^2\bigg/\sigma^2_{\widehat\beta_1}}{\frac{\sum e_t^2}{\sigma^2}\bigg/(n-2)}\notag \\ &=\frac{\chi^2_{(1)}\bigg/1}{\chi^2_{(n-2)}\bigg/(n-2)}\notag\\ &=F_{(1,n-2)} \tag{2.48} \end{align}\]
This result means that the two tests in the simple linear regression leads to the same result about the null hypothesis so that if the computed fisher is equal or less than tabulated fisher (with predefined significance level \(\alpha\)) \(F^c_{(1,n-2)}\leqslant F^{\alpha}_{(1,n-2)}\) we should not reject the null hypothesis.
Testing the overall significance in the simple linear regression reduces to testing the slop significance stated by \(H0:\beta_1=0\). using the above formula (2.47) thus, the computed fisher is:
\[\begin{align} F^c_{(n-2)}&=\frac{\left(\widehat\beta_1-\overbrace{\beta_1}^{H0:\beta_1=0}\right)\sum \left(X_t-\overline X\right)^2}{\sum e_t^2\bigg/(n-2)}\notag\\ &=\frac{\widehat\beta_1\sum \left(X_t-\overline X\right)^2}{\sum e_t^2\bigg/(n-2)}\notag\\ &=\frac{\sum \left(\widehat Y_t-\overline Y\right)^2}{\sum e_t^2\bigg/(n-2)}\notag\\ F^c_{(n-2)}&=\frac{SSE\bigg/1}{SSR\bigg/(n-2)} \tag{2.49} \end{align}\]
It can also expressed in terms of the coefficient of determination \(R^2\) as follows:
\[\begin{align*} F^c_{(n-2)}&=\frac{SSE\bigg/1}{SSR\bigg/(n-2)}\\ &=\frac{SSE\bigg/SST}{SSR\bigg/SST(n-2)}\\ &=\frac{R^2}{(1-R^2)\bigg/(n-2)} \end{align*}\]
Definition 2.13 In the multiple linear model with \(k\) parameters (\((k-1)\) regressors and the constant term), the fisher test has two degrees of freedom, the first one equals to the number of the regressors included in the model \((k-1)\), and the second one equals to \((n-k)\) so that the general formula of fisher test is:
\[\begin{equation*} F^c_{(n-2)}=\frac{SSE\bigg/(k-1)}{SSR\bigg/(n-k)} \end{equation*}\]
because usually, we get this value from statistic tables↩︎