04-Hypothesis Test & p-value

Hypothesis Test Process

1. Set a Null Hypothesis

The null hypothesis can be thought of as the opposite of the "guess" the research made (in this example the biologist thinks the plant height will be different for the fertilizers).  So the null would be that there will be no difference among the groups of plants.

2. Set the significance level (alpha) 

	H0 is true	H0 is false
Accept	true positive	false positive
Reject	true negative (alpha)	false negative

The term significance level (alpha) is used to refer to a pre-chosen probability, also called Type I error.

3. Collect Data

Usually we use Permutation sampling to collect the new random data for the following hypothesis test, called hypothesis data. The original data is called observed data.

4. Calculate the  test statistic 

Could be the mean of differences, pearson correlation r value and so on. Depend on real situation.

5. Calculate the p-value

the term "p-value" is used to indicate a probability that you calculate after a given study. p = "sum(the situations that can prove the null hypothesis is true)/#ofAllSituations"

If your p-value is less than the chosen significance level then you reject the null hypothesis.

A full example:

Null hypothesis: performance in the finals and semifinals are identical 

Test statistic:  the mean of the fractional improvement. mean(f) where f = (semi_perm - final_perm) / semi_perm

f_mean is calculated from observed data, perm_reps[i] is calculated from hypothesis data.

p-value: sum(perm_reps >= f_mean) / len(perm_reps)

so, if hypothesis is true, the ideal situation is that f_mean is some value close to the average value  of perm_reps so that the p-value should be 0.5;

however, the result shows that p-value equals to 0.001, which means that "f_mean" is not gonna happen in this hypothesis. Hence we reject the hypothesis.