Before you select the best model based on your favorite goodness of fit statistic – Mean Squared Error, Gini, K-S, AUC, or misclassification rate – STOP! Model performance metrics are not a one size fits all measure. As an analyst, selecting the right performance metric might mean the difference between having an exceptionally good result, and having no result.
The classic example: There is only a 3% prevalence of the event of interest in my data. I can build a model that is 97% accurate (3% error rate) that NEVER detects the event of interest! In fact, I don’t even need to build a model to get this result – I can just guess “No” 100% of the time.
CHeck out my blog post at: https://communities.sas.com/docs/DOC-2501