Continuing my thoughts on random experiments and what can go wrong:
One common problem is Contamination of the Control Group
The only difference between the treatment and control groups should be the treatment. That said, this isn't always true. How the treatment is administered can affect the control group. Think about health examples where, for example, deworming programs for the treatment group can also impact the control group. Or education examples where the treatment and control groups may receive similar programs in spite of efforts to keep them separate.
If the control group becomes contaminated, the effect size you measure will likely be reduced. While there are analytic methods to address this (such as modeling the degree of exposure or contamination), the best approach is to focus on the study design and implementation so as to avoid this issue whenever possible.
#randomization #statistics #datascience #programevaluation