Any parameter estimation requires statistical hypothesis testing. The point of such a test is to assess whether a given parameter is significant or not given the initial hypothesis.
I. Definition and context
Let’s suppose we are interested in the following problem. We want to assess whether the demand for ice cream depends on the outside temperature. The model we would build would look like this :
- is the icecream demand on a given day
- a constant parameter
- the parameter that assesses the impact of temperature on icecream demand
- the average temparature for a given day
- the residuals
II. Compute the expectation and the variance
By OLS method, we build an estimator for and . We’ll call these estimators and , which can be described by three metrics :
- an expectation
- a bias
- a variance
The expectation of the parameter corresponds… to its expected value. Nothing really new here :
The bias corresponds to how far we are from the actual value. It is given by : . The true bias is typically unknown, as we try to estimate . If the bias is 0, we say that the estimator is unbiased.
The variance defines the stability of our estimator regarding the observations. Indeed, the features might be highly spread, which would mean a pretty big variance.
It can be shown that the variance of is given by :
And the one of by :
Where the estimated variance is defined by : . This is an unbiaised estimator of the variance.
III. Statistical hypothesis testing
1. Two-sided tests
T-Stat and critical value
For each parameter, we want to test whether the parameter in question has a real impact on the output or not, to avoid adding dimensions that bring no significant information. In the linear regression : , it would mean testing whether the Betas are significantly different from 0 or not.
To do so, we proceed to a statistical test. If our aim is the state if the parameter is significantly different from 0, we are doing a test with :
- the null hypothesis :
- and the alternative hypothesis : .
Some further theory is needed here : Recall the Central Limit Theorem. converges to as n tends to infinity if is knowm.
In case is unknown, Slutsky’s Lemma states that converges to if converges to .
Most of the time, is unknown. From this point, it can be shown that : where and is the degrees of freedom (p is the dimension, equal to 1 here).
This metric is called the T-Stat, and it allows us to perform a hypothesis test. The 0 in the numerator can be replaced by any value we would like to test actually.
The T-Stat can be decomposed this way :
- is the estimated parameter
- is the value of the true parameter we are testing
- follows a Normal distribution
- follows a Chi Square Distribution
- a ratio of a Normal over the square root of a Chi Square is a Student distribution
How to interpret the T-Stat?
The T-Stat should be compared with the Critical Value. The critical value is the quantile of the corresponding Student distribution at a given level of . If a coefficient is significant at a level , this means that the T-Stat is above or under the quantiles of the Student Distribution.
A parameter is said to be significant if its value is significantly different from 0, i.e is larger than the critical value.
If the T-Stat is smaller than the critical value, we cannot reject the null hypothesis .
Another interpretation is that the probability that the coefficient estimate is not in the interval is smaller than . This probability is called the p-value and is defined by :
Using the CLT, one can set a confidence interval around an estimate of a parameter.
The lower bound and the upper bound are determined by the critical value of the student distribution at a level , and by the standard deviation of the parameter.
2. One-sided tests
Implicitely, when we defined the hypothesis to test :
- Null Hypothesis the null hypothesis :
- and the alternative hypothesis :
we implied a bi-lateral test. Indeed, when we define , we state that both a negative or a positive value of the parameter are considered as failures of .
Now, let’s redefine the hypothesis :
- Null Hypothesis
In this case, we are only interested in one side of the distribution :
III. Joint hypothesis
Let’s add some new variables to our model to explain and predict the icecreams consumption.
1. One parameter against another
Now, what happens if we want to test one parameter against another?
For example, we could define the new hypothesis this way :
- Null Hypothesis
The T-Stat would become :
The standard deviation is however quite difficult to estimate. For this reason, we define a new parameter : . This way, we can just redefine our hypothesis :
- Null Hypothesis
We can replace in our model :
2. Multiple restrictions
We might also want to test several hypothesis at once. For example, in a more complex model :
We might want to test the following joint hypothesis :
- Null Hypothesis
- and is not true
The one thing is to avoid using individual student tests. The intuition we rather use is the following: if all the coefficients are not jointly significant, the sum of the squared errors should not diminish if we delete some variables. In other words, under a constrained model where , the Sum of Squared Residuals does not change.
Recall that we define the Sum of Squared Residuals (SSR) as :
The constrained model would become :
To test the hypothesis defined above, we define the F-Stat :
- is the sum of squared residuals for the constrained model
- is the sum of squared residuals for the unconstrained model
- is the number of restrictions we apply
Under those hypothesis, the F-Stat follows a Fisher distribution :
We can use the Fisher distribution to find the critical value . As before, we reject if .
If we apply only one constraint, the F-Stat is similar to the T-Stat.
**Conclusion **: We have covered the most common tests to apply in the linear regression framework. Don’t hesitate to drop a comment if you have a question.
Like it? Buy me a coffee