I use SAS JMP to run multivariate testing experiments (MVT, regression), it generates a test plan and performs regression analysis, defining significant effects. The most important thing is to be able to interpret and know the pitfalls.
Mathematical calculations, as such, are not needed.
I have read statistics books such as Montgomery Design of experiments and they are usually full of formulas.
I still cannot understand how are they used in real world.
Yea I understand that I need to know concepts, different types of designs, basics of statistics as p value, standard deviation, how to calculate proper sample size, etc.
I speak about design of experiments mostly in web marketing and user experience fields.
Michael, thank you very very much for your comprehensive answer.
I am not a statistician as well, I have degree in psychology of management and experience in conversion optimisation using design of experiments.
I now see some jobs as "Digital Optimisation Analyst" that look very appealing to me, because they include mainly Design of Experiments in their responsibilities and I want to prepare myself for them. I am quite passionate about DOE.
You say: "Then you are in the world of Exploratory Data Analysis".
For DOE in lets say Direct-Mail you define several factors of mail piece(letter,envelope, flyer; their characteristics, color, call to action, discounts, content, etc), 2 level Yes/No.
You then create different runs=mail pieces and randomly send them out. Depending on conversion rate you can estimate a sample size.
Thank you for your explanation how to view OUTLIERS and test hypothesis with large sample sizes. It is very helpful.