Before you select the best model based on your favorite goodness of fit statistic – Mean Squared Error, Gini, K-S, AUC, or misclassification rate – STOP!  Model performance metrics are not a one size fits all measure.  As an analyst, selecting the right performance metric might mean the difference between having an exceptionally good result, and having no result.   

The classic example:  There is only a 3% prevalence of the event of interest in my data. I can build a model that is 97% accurate (3% error rate) that NEVER detects the event of interest!    In fact, I don’t even need to build a model to get this result – I can just guess “No” 100% of the time. 

CHeck out my blog post at: https://communities.sas.com/docs/DOC-2501