1.3 Inference

In the framework of a univariate linear regression model, one can be interested in testing two different groups of hypotheses about $ \beta $, $ \alpha $ and $ \sigma ^2$. In the first group, the user has some prior knowledge about the value of $ \beta $, for example he believes $ \beta=\beta_0$, then he is interested in knowing whether this value, $ \beta_0$, is compatible with the sample data. In this case the null hypothesis will be $ H_0 \ : \ \beta=\beta_0$, and the alternative $ H_1 \ : \ \beta \ne \beta_0$. This is what is called a two sided test. In the other group, the prior knowledge about the parameter $ \beta $ can be more diffuse. For example we may have some knowledge about the sign of the parameter, and we want to know whether this sign agrees with our data. Then, two possible tests are available, $ H_0 \ : \
\beta\le\beta_0$ against $ H_1 \ : \ \beta > \beta_0$, (for $ \beta_0=0$ this would be a test of positive sign); and $ H_0 \ : \
\beta\ge\beta_0$ against $ H_1 \ : \ \beta < \beta_0$, (for $ \beta_0=0$ this would be a test of negative sign). These are the so called on sided tests. Equivalent tests for $ \alpha $ are available.

The tool we are going to use to test for the previous hypotheses is the sampling distribution for the different estimators. The key to design a testing procedure lies in being able to analyze the potential variability of the estimated value, that is, one must be able to say whether a large divergence between it and the hypothetical value is better ascribed to sampling variability alone or whether it is better ascribed to the hypothetical value being incorrect. In order to do so, we need to know the sampling distribution of the parameters.


1.3.1 Hypothesis Testing about $ \beta $

In section 1.2.6, equations (1.52) to (1.58) show that the joint finite sample distribution of the OLS estimators of $ \alpha $ and $ \beta $ is a normal density. Then, by standard properties of the multivariate gaussian distribution (see Greene (1993), p. 76), and under assumptions (A.1) to (A.7) from Section (1.2.6)it is possible to show that

$\displaystyle \hat\beta \sim {\bf N}\left(\beta, \frac{\sigma^2}{\sum^n_{i=1}\left(x_i-\bar{x}\right)^2}\right),$ (1.63)

then, by a standard transformation

$\displaystyle z=\frac{\hat\beta-\beta}{\sqrt{\sigma^2/\sum_{i=1}^n(x_i-\bar x)^2}}$ (1.64)

is standard normal. $ \sigma ^2$ is unknown and therefore the previous expression is unfeasible. Replacing the unknown value of $ \sigma ^2$ with $ \hat\sigma^2$ (the unbiased estimator of $ \sigma ^2$) the result

$\displaystyle z=\frac{\hat\beta-\beta}{\sqrt{\hat\sigma^2/\sum_{i=1}^n(x_i-\bar x)^2}},$ (1.65)

is the ratio of a standard normal variable (see (1.63)) and the square root of a chi-squared variable divided by its degrees of freedom (see (1.61)). It is not difficult to show that both random variables are independent, and therefore $ z$ in (1.65) follows a student-t distribution with $ n-2$ degrees of freedom (see Johnston and Dinardo (1997), p. 489 for a proof). i. e.

$\displaystyle z \sim t_{(n-2)}$ (1.66)

To test the hypotheses, we have the following alternative procedures:

  Null Hypothesis Alternative Hypothesis
a) Two-sided test H$ _{0}: \beta = \beta_{0}$ H$ _{1}: \beta \neq \beta_{0}$
b) one-sided test    
Right-sided test H$ _{0}: \beta \leq \beta_{0}$ H$ _{1}: \beta
> \beta_{0}$
Left-sided test H$ _{0}: \beta \geq \beta_{0}$ H$ _{1}: \beta <
\beta_{0}$

According to this set of hypotheses, next, we present the steps for a one-sided test, after this, we present the procedure for a two-sided test.

One-sided Test

The steps for a one-sided test are as follows:

Step 1:
Establish the set of hypotheses

H$\displaystyle _{0}: \beta \leq \beta_{0}$    versus    H$\displaystyle _{1}: \beta > \beta_{0}.$

Step 2:
The test statistic is $ \frac{\hat\beta-\beta_0}{\sqrt{\hat\sigma^2/\sum_{i=1}^n(x_i-\bar
x)^2}}$, which can be calculated from the sample. Under the null hypothesis, it has the t-distribution with $ (n-2)$ degrees of freedom. If the calculated $ z$ is "large", we would suspect that $ \beta $ is probably not equal to $ \beta_0$. This leads to the next step.
Step 3:
In the $ t$-table, look up the entry for $ n-2$ degrees of freedom and the given level of significance $ (\epsilon)$ and find the point $ t^*_{\epsilon, n-2}$ such that $ P(t>t^*)=\epsilon$
Step 4:
Reject $ H_0$ if $ z>t^*_{\epsilon,n-2}$.

If the calculated $ t$-statistic $ (z)$ falls in the critical region, then $ z>t^*_{\epsilon,n-2}$. In that case the null hypothesis is rejected and we conclude that $ \beta $ is significantly greater than $ \beta_0$

The $ p$-value Approach to Hypothesis Testing

The $ t$-statistic can also be carried out in an equivalent way. First, calculate the probability that the random variable $ t$ ($ t$-distribution with $ n-2$ degrees of freedom) is greater than the observed $ z$, that is, calculate

$\displaystyle p-value=P(t>z)$

This probability is the area to the right of $ z$ in the $ t$-distribution. A high value for this probability implies that the consequences of erroneously rejecting a true $ H_0$ is severe. A low $ p$-value implies that the consequences of rejecting a true $ H_0$ erroneously are not very severe, and hence we are "safe" in rejecting $ H_0$. The decision rule is therefore to "accept" $ H_0$(that is, not reject it) if the $ p$-value is too high. In other words, if the $ p$-value is higher than the specified level of significance (say $ \epsilon$), we conclude that the regression coefficient $ \beta $ is not significantly greater than $ \beta_0$ at the level $ \epsilon$. If the $ p$-value is less than $ \epsilon$ we reject $ H_0$ and conclude that $ \beta $ is significantly greater than $ \beta_0$. The modified steps for the $ p$-value approach are as follows:

Step 3a:
Calculate the probability (denoted as $ p$-value) that $ t$ is greater than $ z$, that is, compute the area to the right of the calculated $ z$.
Step 4a:
Reject $ H_0$ and conclude that the coefficient is significant if the $ p$-value is less than the given level of significance $ (\epsilon)$

If we want to establish a more constrained null hypothesis, that is, the set of possible values that $ \beta $ can take under the null hypothesis is only one value, we must use a two-sided test.

Two-sided Test

The procedure for a two-sided alternative is quite similar. The steps are as follows:

Step 1:
Establish the set of hypotheses

H$\displaystyle _{0}: \beta = \beta_{0}$    versus    H$\displaystyle _{1}: \beta \neq \beta_{0}.$

Step 2:
The test statistic is $ \frac{\hat\beta-\beta}{\sqrt{\hat\sigma^2/\sum_{i=1}^n(x_i-\bar
x)^2}}$, which is the same as before. Under the null hypothesis, it has the t-distribution with $ (n-2)$ degrees of freedom.
Step 3:
In the $ t$-table, look up the entry for $ n-2$ degrees of freedom and the given level of significance $ (\epsilon)$ and find the point $ t^*_{\epsilon /2, n-2}$ such that $ P(t>t^*)=\epsilon /2$ (one-half of the level of significance)
Step 3a:
To use the $ p$-value approach calculate

$\displaystyle p-value=P(t>z$   or$\displaystyle \quad t<z)=2P(t>z)$

because of the symmetry of the $ t$-distribution around the origin.

Step 4:
Reject $ H_0$ if $ \vert z\vert>t^*_{\epsilon /2,n-2}$ and conclude that $ \beta $ is significantly different form $ \beta_0$ at the level $ \epsilon$
Step 4a:
In case of the $ p$-value approach, reject $ H_0$ if $ p$-value$ <\epsilon$, the level of significance.

The different sets of hypotheses and their decision regions for testing at a significance level of $ \epsilon$ can be summarized in the following table:

Test Rejection region for H$ _{0}$ Non-rejection region for H$ _{0}$
Two-sided $ \left\{ z\,\vert\,
z<-t^*_{\epsilon /2}\, \text{ or } \,z>t^*_{\epsilon /2}\right\}$ $ \left\{ z\,\vert\, -t^*_{\epsilon /2}\leq z \leq t^*_{\epsilon
/2}\right\}$
right-sided $ \left\{ z\,\vert\,
z>t^*_{\epsilon}\right\}$ $ \left\{ z\,\vert\, z\leq
t^*_{\epsilon}\right\}$
left-sided $ \left\{ z\,\vert\,
z<-t^*_{\epsilon}\right\}$ $ \left\{ z\,\vert\, z\geq
-t^*_{\epsilon}\right\}$


1.3.2 Example

We implement the following Monte Carlo experiment. We generate one sample of size n = 20 of the model $ y_i = 2+0.75 x_i+u_i \quad
i=1,\ldots,20$. $ X$ has a uniform distribution generated as follows $ X\sim U[0,1]$, and the error term $ u\sim \textrm{N}(0,1)$. We estimate $ \alpha $, $ \beta $, $ \sigma ^2$. The program gives the three possible test for $ \beta $ when $ \beta_0=0$, showing the critical values and the rejection regions.

4390 XEGlinreg13.xpl

The previous hypothesis-testing procedure is confined to the slope coefficient, $ \beta $. In the next section we present the process based on the fit of the regression


1.3.3 Testing Hypothesis Based on the Regression Fit

In this section we present an alternative view to the two sided test on $ \beta $ that we have developed in the previous section. Recall that the null hypothesis is $ H_0 \ : \ \beta=\beta_0$ against the alternative hypothesis that $ H_0 \ : \beta \ne \beta_0$.

In order to implement the test statistic remind that the OLS estimators, $ \hat\beta$ and $ \hat\alpha$, are such that they minimize the residual sum of squares (RSS). Since $ R^2=1-RSS/TSS$, equivalently $ \hat\beta$ and $ \hat\alpha$ maximize the $ R^2$, and therefore any other value of $ \hat\beta$, leads to a relevant loss of fit. Consider, now, the value under the null, $ \beta_0$ rather than $ \hat\beta$ (the OLS estimator). We can investigate the changes in the regression fit when using $ \beta_0$ instead of $ \hat\beta$. To this end, consider the following residual sum of squares where $ \hat\beta$ has been replaced by $ \beta_0$.

$\displaystyle RSS_0 = \sum^n_{i=1}(y_i-\alpha-\beta_0 x_i)^2.$ (1.67)

Then, the value of $ \alpha $, $ \alpha_0$, that minimizes (1.67) is

$\displaystyle \alpha_0=\bar y-\beta_0 \bar x.$ (1.68)

Substituting (1.68) into (1.67) we obtain

$\displaystyle RSS_0 = \sum^n_{i=1}\left(y_i-\bar y-\beta_0 (x_i-\bar{x})\right)^2.$ (1.69)

Doing some standard algebra we can show that this last expression is equal to

$\displaystyle RSS_0 = TSS + \left(\hat\beta-\beta_0\right)^2\sum^n_{i=1}(x_i-\bar{x})^2 - ESS,$ (1.70)

and since $ TSS = ESS + RSS$ and defining

$\displaystyle R^2_0=1-\frac{RSS_0}{TSS}$ (1.71)

then (1.70) is equal to

$\displaystyle R^2-R^2_0=\frac{(\hat\beta-\beta_0)^2\sum_{i=1}^n(x_i-\bar x)^2}{TSS},$ (1.72)

which is positive, because $ R^2_0$ must be smaller than $ R^2$, that is, the alternative regression will not fit as well as the OLS regression line. Finally,

$\displaystyle F = \frac{(R^2-R^2_0)/1}{(1-R^2)/(n-2)} \sim \digamma_{1,n-2}$ (1.73)

where $ \digamma_{1,n-2}$ is an F-Snedecor distribution with $ 1$ and $ n-2$ degrees of freedom. The last statement is easily proved since under the assumptions established in Section 1.2.6 then

$\displaystyle (\hat\beta-\beta_0)^2\sum_{i=1}^n(x_i-\bar x)^2/\sigma^2 \sim \chi^2_1,$ (1.74)

$\displaystyle (n-2)RSS/\sigma^2 \sim \chi^2_{n-2},$ (1.75)

and

$\displaystyle \frac{(R^2-R^2_0)/1}{(1-R^2)/(n-2)}= \frac{(\hat\beta-\beta_0)^2\sum_{i=1}^n(x_i-\bar x)^2/\sigma^2}{(n-2)RSS/\sigma^2}.$ (1.76)

The proof of (1.73) is closed by remarking that (1.74) and (1.75) are independent.

The procedure in the two-sided test

Step 1:
Establish the set of hypotheses

H$\displaystyle _{0}: \beta = \beta_{0}$    versus    H$\displaystyle _{1}: \beta \neq \beta_{0}.$

Step 2:
The test statistic is $ F=\frac{(R^2-R^2_0)/1}{(1-R^2)/(n-2)}$. Under the null hypothesis, it has the F-distribution with one and $ (n-2)$ degrees of freedom.

Step 3:
In the $ F$-table, look up the entry for $ 1,n-2$ degrees of freedom and the given level of significance $ (\epsilon)$ and find the point $ \digamma^*_{\epsilon /2,1,n-2}$ and $ \digamma^*_{1-\epsilon /2,1,n-2}$

Step 4:
Reject $ H_0$ if $ F_0>\digamma^*_{\epsilon /2,1,n-2}$ or $ F_0<\digamma^*_{1-\epsilon /2,1,n-2}$ and conclude that $ \beta $ is significantly different from $ \beta_0$ at the level $ \epsilon$


1.3.4 Example

With the same data of the previous example, the program computes the hypothesis test for $ H_0: \beta_0=0$ by using the regression fit. The output is the critical value and the rejection regions.

4636 XEGlinreg14.xpl


1.3.5 Hypothesis Testing about $ \alpha $

As in Section 1.3.1, by standard properties of the multivariate gaussian distribution (see Greene (1993), p. 76), and under assumptions (A.1) to (A.7) from Section (1.2.6) it is possible to show that

$\displaystyle z=\frac{\hat\alpha-\alpha}{\hat\sigma \sqrt{1/n+\bar x^2/\sum_{i=1}^n(x_i-\bar x)^2}} \sim t_{(n-2)}$ (1.77)

The construction of the test are made similar to the test of $ \beta $, a two- or one-sided test will be carried out:

1)Two-sided test

H$\displaystyle _{0}: \alpha = \alpha_{0}$    versus    H$\displaystyle _{1}: \alpha \neq \alpha_{0}.$

2) Right-sided test

H$\displaystyle _{0}: \alpha \leq \alpha_{0}$    versus    H$\displaystyle _{1}: \alpha > \alpha_{0}.$

3) Left-sided test

H$\displaystyle _{0}: \alpha \geq \alpha_{0}$    versus    H$\displaystyle _{1}: \alpha < \alpha_{0}.$

If we assume a two-sided test, the steps for this test are as follows

Step 1:
Establish the set of hypotheses

H$\displaystyle _{0}: \alpha = \alpha_{0}$    versus    H$\displaystyle _{1}: \alpha \neq \alpha_{0}.$

Step 2:
The test statistic is $ z=\frac{\hat\alpha-\alpha_0}{\hat\sigma_{\hat\alpha}}$, which is the same as before. Under the null hypothesis, it has the t-distribution with $ (n-2)$ degrees of freedom.
Step 3:
In the $ t$-table, look up the entry for $ n-2$ degrees of freedom and the given level of significance $ (\epsilon)$ and find the point $ t^*_{\epsilon /2, n-2}$ such that $ P(t>t^*)=\epsilon /2$ (one-half of the level of significance)
Step 4:
Reject $ H_0$ if $ \vert z\vert>t^*_{\epsilon /2,n-2}$ and conclude that $ \alpha $ is significantly different form $ \alpha_0$ at the level $ \epsilon$


1.3.6 Example

With the same data of the previous example, the program gives the three possible tests for $ \hat\alpha$ when $ \alpha_0=2$, showing the critical values and the rejection regions.

4760 XEGlinreg15.xpl


1.3.7 Hypotheses Testing about $ \sigma ^2$

Although a test for the variance of the error term $ \sigma ^2$ is not as common as one for the parameters of the regression line, for the sake of completeness we present it here. The test on $ \sigma ^2$ can be obtained from the large sample distribution of $ \sigma ^2$,

$\displaystyle \frac{(n-2)\hat\sigma^2}{\sigma^2}\sim \chi_{n-2}^2$ (1.78)

Using this result, one may write:

$\displaystyle {\rm Prob}\left[\chi^2_{1-\epsilon /2} < \frac{(n-2)\hat\sigma^2}{\sigma^2} < \chi^2_{\epsilon /2}\right] = 1-\epsilon$ (1.79)

which states that $ \epsilon$ percent of the values of a $ \chi^2$ variable will lie between the values that cut off $ \epsilon /2$ percent in each tail of the distribution. The critical values are taken from the $ \chi^2$ distribution with $ (n-2)$ degrees of freedom. Remember that the $ \chi^2$ is an asymmetric distribution.

The $ (1-\epsilon)$ percent confidence interval for $ \sigma ^2$ will be:

$\displaystyle \left(\frac{(n-2)\hat\sigma^2}{\chi^2_{\epsilon /2}}; \frac{(n-2)\hat\sigma^2}{\chi^2_{1-\epsilon /2}}\right)$ (1.80)

Now, similar to test the coefficients of the regression, we can consider a test for the significance of the error variance $ \sigma ^2$. The steps are as follows:

Step 1:
Establish the set of hypotheses

H$\displaystyle _{0}: \sigma^2 = \sigma^2_{0}$    versus    H$\displaystyle _{1}: \sigma^2 \neq \sigma^2_{0}.$

Step 2:
The test statistic is $ q=(n-2)\frac{\hat\sigma^2}{\sigma^2_0}$. The distribution of this, under the null hypothesis, is chi-squared with $ (n-2)$ degrees fo freedom. If $ q$ is "large" we would suspect that $ \sigma ^2$ is probably not equal to $ \sigma^2_0$
Step 3:
From the chi-square table, look at the value $ \chi^{2}_{\epsilon /2,n-2}$ and $ \chi^{2}_{1-\epsilon /2,n-2}$
Step 4:
We reject $ H_0$ if the value of the statistic $ q\geq\chi^2_{\epsilon /2}$ or $ q\leq\chi^2_{1-\epsilon /2}$. Otherwise, $ H_0$ can't be rejected. This means that $ H_0$ is accepted if $ \sigma^2_0$ lay in the confidence interval of $ \sigma ^2$ (Chow, 1983).