Every product team wants to know what makes their product thrive. They want to know how to optimize metrics and leave users the happiest and most engaged. Without a way to definitively understand user behavior, they must turn to anything they can. Enter the A/B test.

Ultimately, every A/B test starts from a hypothesis. The hypothesis could be, “If we did [blank], then we would improve conversion.” Or “Feature X should drive increased retention, let’s test out that assumption through an A/B test.”

The goal is almost always to drive a KPI such as conversion, retention, engagement, etc. These are lag measures, meaning that they result from a change in a lead measure that the product team can control.

If the Product team thinks that engaging with Feature X boosts retention, they can set up a test where half of new users see Feature X and half do not.

However, it is important to note there is a business risk to adding a feature that could hurt existing KPIs. Therefore, teams often introduce a new feature to one percent of users and ensure that it did not do any harm. From there, the team “ramps up” the percent of users seeing the new feature.

When starting a test, it’s good practice to know how long it will take to reach statistical significance based on the size of each test group. There are various tools that make this fairly simple to estimate. For this example, let’s assume it’ll take 14 days of data to reach significance.

Once those 14 days have passed, it’s time to look at the readout of the test.

Using this readout, we can do a z-test to see if there a statistically significant difference between the control and test groups. We do a two-sided test: this means that we assume the two groups are equal unless that is proven to be very unlikely by our data, at which point we assume there is a difference. We can plug our data into a significance calculator, and we find statistical significance.

We have reason to believe that Feature X did in fact boost retention for new users. In this case, we should roll out this feature for all users.

But what if this was not the case? What if it did not drive a retention increase? In that case, it may be because the retention rates are too close to call, and the test may need more time to gather data.

It’s entirely possible that there never will a significant outcome from test, and that’s okay. Then it’s time to go back to the drawing board. The Product team will have to conduct a post-mortem on why this feature did not achieve its goals.

Then comes another hunch, another hypothesis, and the process comes to life again.

* *

© 2019 Data Science Central ® Powered by

Badges | Report an Issue | Privacy Policy | Terms of Service

**Most Popular Content on DSC**

To not miss this type of content in the future, subscribe to our newsletter.

- Book: Statistics -- New Foundations, Toolbox, and Machine Learning Recipes
- Book: Classification and Regression In a Weekend - With Python
- Book: Applied Stochastic Processes
- Long-range Correlations in Time Series: Modeling, Testing, Case Study
- How to Automatically Determine the Number of Clusters in your Data
- New Machine Learning Cheat Sheet | Old one
- Confidence Intervals Without Pain - With Resampling
- Advanced Machine Learning with Basic Excel
- New Perspectives on Statistical Distributions and Deep Learning
- Fascinating New Results in the Theory of Randomness
- Fast Combinatorial Feature Selection

**Other popular resources**

- Comprehensive Repository of Data Science and ML Resources
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- 100 Data Science Interview Questions and Answers
- Cheat Sheets | Curated Articles | Search | Jobs | Courses
- Post a Blog | Forum Questions | Books | Salaries | News

**Archives:** 2008-2014 |
2015-2016 |
2017-2019 |
Book 1 |
Book 2 |
More

**Most popular articles**

- Free Book and Resources for DSC Members
- New Perspectives on Statistical Distributions and Deep Learning
- Time series, Growth Modeling and Data Science Wizardy
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- Comprehensive Repository of Data Science and ML Resources
- Advanced Machine Learning with Basic Excel
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- Selected Business Analytics, Data Science and ML articles
- How to Automatically Determine the Number of Clusters in your Data
- Fascinating New Results in the Theory of Randomness
- Hire a Data Scientist | Search DSC | Find a Job
- Post a Blog | Forum Questions

## You need to be a member of Data Science Central to add comments!

Join Data Science Central