I want to test the optimum price for some items sold online. One way to do it is to set two different prices and do some A/B testing to see which price generates the most revenue, or comparing user-customized versus flat prices, using Thompson sampling, the Taguchi method or multi-armed A/B testing.
How to proceed if you want to test a continuous set of prices, not just two or three prices A / B / C? Is testing (say) 10,000 different prices any better than standard A/B testing, or does it lead to over-fitting and thus a non-robust solution? Likewise, if you want to test which background color works best for a website, is testing one million different colors more efficient than standard testing, and how to do it?
Also, let's say you want to modify 20 features on your website, each one having 4 potential values (color, font size, font face and so on). In short, instead of A/B testing with 2 potential outcomes (A or B), you perform a multivariate test with 4^20 (4 at power 20) outcomes. Of course you will be able to test only a tiny fraction of all the possibilities, but is it more efficient than sequentially doing an A/B test for one feature, then another A/B test for another feature, and so on? The latter approach would take a lot of time and would result in a very local optimum. For instance, for the first feature, maybe A works best, for the second one (after choosing A for the first one) C works best, but for both featured combined, maybe (D, B) works best. How to do such a test when the number of potential combinations is 4^20?
Finally, how do you determine the sample size for these types of experiments? Or in other words, what is the stopping criterion? Are p-values still being used in this context?
The technique used by leading market research professionals to solve the problems you are describing is called conjoint or discrete choice analysis. Here is a brief example of the type of problem conjoint analysis helps solve https://www.sawtoothsoftware.com/download/techpap/undca15.pdf. Conjoint analysis helps you develop the preferred product or service based on almost an unlimited number of attributes (in your examples, price, color, font size, etc.) and levels (different prices, different colors, different fonts sizes, etc.). The ultimate output of the conjoint model is a simulator that lets you test for the best product (based on market share, sales, other objectives) and can provide you a price elasticity of demand curve for a price continuum as you describe. You may want to look at the Sawtooth Software site for more information about this technique https://www.sawtoothsoftware.com/ (they use Hierarchical Bayesian Regression to drive their models). Sawtooth Software's software is very flexible but does take some learning to get up to speed on using. AYTM https://aytm.com/ is a DIY platform that has a more limited self service conjoint offering. If you want to work with a consultant, I 'm a big fan of Megan Peitz at Numerious (worked with her last year on something). She is smart, driven, passionate and a pleasure to work with. Here is her email [email protected] and Linked-in profile linkedin.com/in/meganpeitz. I hope that is helpful. Good luck.