Why have a control group?
In a
properly designed A/B test:
Subjects are
randomized
to either treatments
Need to collect data on treatments A and B in such a way that any observed difference between A and B must be due to either:
Random chance in assignment of subjects
A true difference between A and B
Must establish a single test statistic before conducting the test
, otherwise leave room for researcher bias
Why just A/B? Why not C, D…?
If testing which is the best out of multiple possible conditions, use