Comment

Comment by Michael Morgan on January 22, 2020 at 4:51am

I can offer a couple of observation on this question that may help data scientists understand the most immediate needs for understanding the need to understand just two basic mathematical concepts.  These are not difficult to understand intuitively, but in data science, and in neural networks specifically, we have changed the statistical names to terms that data scientists, and computer scientists in particular, can use as handles for problems we run into.  In a new blog, I’ll trace these in recent data science work, but here to summarize:

1.  Vanishing gradient.  This is the main reason for ongoing improvements and tweaks to NNs.  Especially problems emerging when trying to backpropogqte from a near-vanishing gradient,  The true cause of this persistent problem Is multicollinearity among parameters.  It is a statistical problem, but not hard for those not trained in multivariate statistics to understand.  Knowing this can help the NN analyst to choose better mini-interventions than those currently popular.

2.  Too many parameters.  Even if multicollinearity is controlled through the right preprocessing of the data, or perhaps some sort of masking to avoid letting the model continue to use data points that have already served up their information, the number of parameters can mushroom and preclude sensible inference and/or help with new NN innovations.  The capability to interpret parameters may be secondary to predictive power, but the most successful NN these days appear to incorporate some kind of generative component, such as adversarial tests of intermediate portions of the NN (GAN).

2.