Hypothesis Test

Hypothesis test is a technique where sample data is used to determine if the confidence interval supports a particular claim. Hypothesis tests quantify how likely our data is given a particular claim.

This page will focus on usage of hypothesis tests in the context of mean comparison.

Assumptions for HT

We use the same assumptions used for a confidence interval for the difference in means

1. A random sample was taken
2. Population 1 and 2 are independent
3. Sample means are normally distributed

• • Sample size is large enough (> 30)
  • Population is normally distributed

Procedure (Mean Comparison)

1. Null and Alternative Hypothesis

To perform hypothesis test with mean comparison, we need two things:

• The null hypothesis $H_{0}$ is the statement which we assume to be true
• The alternative hypothesis $H_{A}$ is the complement of the null hypothesis.

Mean comparison work with the difference in means

$\mu _{1}-\mu _{2}$

As such, there are three sets of hypotheses:

• $H_{0}:\mu _{1}-\mu _{2}=0$ vs $H_{A}:\mu _{1}-\mu _{2}\neq 0$
• $H_{0}:\mu _{1}-\mu _{2}\geq 0$ vs $H_{A}:\mu _{1}-\mu _{2}<0$
• $H_{0}:\mu _{1}-\mu _{2}\leq 0$ vs $H_{A}:\mu _{1}-\mu _{2}>0$

2. Test-Statistic

Next, we need to calculate a test-statistic $t_{s}$. This measures how much our sample data differ from $H_{0}$. It summarizes our data to one number to perform hypothesis test on.

For mean comparison, the hypothesized difference is 0 (i.e. the means are the same). Therefore, the test-statistic is calculated as follows:

$t_{s}={\frac {{\bar {y_{1}}}-{\bar {y_{2}}}-0}{\sqrt {{\frac {s_{1}^{2}}{n_{1}}}+{\frac {s_{2}^{2}}{n_{2}}}}}}$

On the numerator, the difference mean is subtracted by 0 to since that is the comparison point; all three sets of hypotheses in mean comparison compares against 0.

On the denominator, the value is divided by the sample standard deviation. This is a surprise tool that will help us later (bottom of 3.)

$t_{s}$, the more our data differs from $H_{0}$. Notice that it increases with sample mean difference and decreases with variance.

3. Find P-value

The p-value is the probability of observing our data or more extreme if $H_{0}$ is in fact true. To find this, we first need to know the sampling distribution of our random variable.

Distribution

In the case of mean comparison, because sample mean has normal distribution, by RV linear combination, the sampling distribution of ${\bar {Y}}_{1}-{\bar {Y}}_{2}$ is

$({\bar {Y_{1}}}-{\bar {Y_{2}}})\sim N(\mu _{1}-\mu _{2},{\frac {\sigma _{1}^{2}}{n_{1}}}+{\frac {\sigma _{2}^{2}}{n_{2}}})$

Both means follow the t-distribution, therefore the difference also follows t-distribution.

We are not going to derive it, but the degree of freedom in this case is

$df=\upsilon ={\frac {({\frac {s_{1}^{2}}{n_{1}}}+{\frac {s_{2}^{2}}{n_{2}}})^{2}}{{\frac {(s_{1}^{2}/n_{1})^{2}}{n_{1}-1}}+{\frac {(s_{2}^{2}/n_{2})^{2}}{n_{2}-1}}}}$

where $\upsilon$ is rounded down when using the t-table.

Remember how the test-statistic has the sample deviation on the denominator? This is so that we can use the t-distribution to calculate the probability! Now that we know the degrees of freedom and the test-statistic to compare against, we can calculate the p-value.

P-value

In the case of mean comparison, we have the following p-values:

• For $H_{A}:\mu _{1}-\mu _{2}\neq 0$, the p-value is $2P(t>|t_{s}|)$
  • Two tails
• For $H_{A}:\mu _{1}-\mu _{2}>0$, the p-value is $P(t>t_{s})$
  • Upper tail
• For $H_{A}:\mu _{1}-\mu _{2}<0$, the p-value is $P(t<t_{s})$
  • Lower tail

The smaller the p-value, the less likely it is to observe our data or more extreme if $H_{0}$ is true, meaning that our data is unlikely if our claim is true.

If there are no calculators, we can use a range of t-values that contain the test-statistic. The result would be a lower and upper bound p-value.

4. Conclusion

We decide a cutoff point for our p-values, typically at $\alpha =0.1,0.05,0.01$, called the level of significance.

If $p<\alpha$, our data supports $H_{A}$, therefore $H_{0}$ is rejected. Otherwise, we failed to reject $H_{0}$.

A CI that covers $0$ implies that there is no significant difference, as it is plausible for the population means to be equal.

Errors

There are two types of possible errors for our conclusion.

• Type I error: Null is true but we rejected it
• Type II error: Null is false but we failed to reject it

Remeber the level of significance? Well it is actually

$\alpha =P(TypeIError)=P(RejectH_{0}|H_{0}True)$

So when setting the level of significance, we are actually controlling type I error. The reason is that the distribution of $t_{s}$ is non-specific: there are lots of possibilities that $H_{0}$ is false.

We also have

$\beta =P(TypeIIError)=P(FTRH_{0}|H_{0}False)$

As $\alpha$ increase, $\beta$ decreases. Therefore, we can choose what $\alpha$ to use depending on which error is probably worse. We cannot control $\beta$. I don't really get why.

If we think Type I error is use, choose smaller $\alpha =0.01$. Otherwise, use $\alpha =0.10$.