What would you do if?
Bayesian Googe Optimize says probability to be best for Original is 64% (sales CR) and Frequentists's calculation says Original is higher that the Test by 10% (sales CR) but the difference is not statistically significant or not enough sample size.
Meanwhile, Test version shows statistical significance in Add to Cart CR only.
Sample size is 200K. We cannot run the test more. Original is a delayed CTA page.
Would you stick with the Original hopping that it will indeed provide 10% boost in sales (64% probability to be best, 10% higher CR but not significant)? Or would you stick with Test version that has (15% higher CR in add to cart and significant)?