Hypothesis tests

Tests whether random chance might be responsible for an observed effect

Intelligence Refinery

The null hypothesis

The baseline assumption that any difference between the groups is due to chance

One way to test the null hypothesis is via

resampling permutation procedure⁠

⁠

Shuffle together the results from groups A and B, then repeatedly deal out the data in groups of similar sizes, then observe how often we get a difference as extreme as the observed difference

The nature of the null hypothesis determines the structure of the hypothesis test

Creates a “null” probability model and tests whether the observed effet is a reasonable outcome of that model

⁠

The alternative hypothesis

The opposite of the null hypothesis, where the difference between groups is due to the treatment

Taken together, the null and alternative hypotheses must account for all possibilities

⁠

One-way, two-way hypothesis test

One-way (one-tail) test

Extreme chance results in only one direction count toward the p-value

Suited for A/B decision making, as one option is assigned as "default” unless the other one proves to be better

Two-way (two-tail) test

Extreme chance results in either direction count toward the p-value

More conservative, many statisticians use it to avoid argument

The null hypothesis

The alternative hypothesis

One-way, two-way hypothesis test

Gallery

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.