In this tutorial, I will give a quick introduction to online experimentation and A/B testing.
The topic requires some basic understanding of Statistics at 101 level. To keep the tutorial self-contained, I will first give an overview of stats fundamentals needed to understand A/B testing. I will explain how A/B testing is done in an online business. In the end, I will mention some of the pitfalls that are common when running online experiments.
The tutorial can roughly be divided into three parts
• Motivation for online experiments. Examples of some A/B tests that defy intuition. Examples from Obama 2012 campaign.
• Statistics fundamentals for hypothesis testing: Confidence intervals, p-values, t-test, sample size, power.
• Art of interpreting results of an online experiment.
• Fallacies and pitfalls.