# Questions about A/B and Multivariate Testing

I want to test the optimum price for some items sold online. One way to do it is to set two different prices and do some A/B testing to see which price generates the most revenue, or comparing user-customized versus flat prices, using Thompson sampling, the Taguchi method  or multi-armed A/B testing.

How to proceed if you want to test a continuous set of prices, not just two or three prices A / B / C? Is testing (say) 10,000 different prices any better than standard A/B testing, or does it lead to over-fitting and thus a non-robust solution? Likewise, if you want to test which background color works best for a website, is testing one million different colors more efficient than standard testing, and how to do it?

Also, let's say you want to modify 20 features on your website, each one having 4 potential values (color, font size, font face and so on). In short, instead of A/B testing with 2 potential outcomes (A or B), you perform a multivariate test with 4^20 (4 at power 20) outcomes. Of course you will be able to test only a tiny fraction of all the possibilities, but is it more efficient than sequentially doing an A/B test for one feature, then another A/B test for another feature, and so on? The latter approach would take a lot of time and would result in a very local optimum. For instance, for the first feature, maybe A works best, for the second one (after choosing A for the first one) C works best, but for both featured combined, maybe (D, B) works best. How to do such a test when the number of potential combinations is 4^20?

Finally, how do you determine the sample size for these types of experiments? Or in other words, what is the stopping criterion? Are p-values still being used in this context?

Views: 944

Capri,