Randomization, including A/B testing, often fails...for a variety of reasons. When that happens you will likely find yourself in a situation where the treatment and control groups differ significantly.
First thing to remember is that some differences can occur in studies that are unavoidable. Are the differences practically significant? Are they statistically significant?
If differences exist that are significant then some options include:
-Prior to intervention, some statisticians recommend re-randomizing
- Can estimate the treatment effect conditional on the un-balanced covariates.
- Can adjust statistically for differences in the groups: for example in a regression model including the factors that are different in the model can reduce some of the bias in the estimate
- Can use quasi-experimental methods to improve the causal inference (propensity methods, instrument variable, interrupted time series, etc.)
Having data that is not randomized makes inference more difficult this is a challenge that can be addressed with some straightforward analysis and data modeling.
#statistics #datascience #randomization #programevaluation