The baseline assumption that any difference between the groups is due to chance

One way to test the null hypothesis is via

resampling permutation procedure

Shuffle together the results from groups A and B, then repeatedly deal out the data in groups of similar sizes, then observe how often we get a difference as extreme as the observed difference

The nature of the null hypothesis determines the structure of the hypothesis test

Creates a “null” probability model and tests whether the observed effet is a reasonable outcome of that model

The alternative hypothesis

The opposite of the null hypothesis, where the difference between groups is due to the treatment

Taken together, the null and alternative hypotheses must account for all possibilities

One-way, two-way hypothesis test

One-way (one-tail) test

Extreme chance results in only one direction count toward the p-value

Suited for A/B decision making, as one option is assigned as "default” unless the other one proves to be better

Two-way (two-tail) test

Extreme chance results in either direction count toward the p-value

More conservative, many statisticians use it to avoid argument