Book: Evaluating Machine Learning Models

Data science today is a lot like the Wild West: there’s endless opportunity and excitement, but also a lot of chaos and confusion. If you’re new to data science and applied machine learning, evaluating a machine-learning model can seem pretty overwhelming. Now you have help. With this O’Reilly report, machine-learning expert Alice Zheng takes you through the model evaluation basics.

In this overview, Zheng first introduces the machine-learning workflow, and then dives into evaluation metrics and model selection. The latter half of the report focuses on hyperparameter tuning and A/B testing, which may benefit more seasoned machine-learning practitioners.

With this report, you will:

• Learn the stages involved when developing a machine-learning model for use in a software application
• Understand the metrics used for supervised learning models, including classification, regression, and ranking
• Walk through evaluation mechanisms, such as hold?out validation, cross-validation, and bootstrapping
• Explore hyperparameter tuning in detail, and discover why it’s so difficult
• Learn the pitfalls of A/B testing, and examine a promising alternative: multi-armed bandits
• Get suggestions for further reading, as well as useful software packages

Get the Free Ebook, here.

About the Author:

Alice Zheng

Alice is a technical leader in the field of Machine Learning. Her experience spans algorithm and platform development and applications. Currently, she is a Senior Manager in Amazon’s Ad Platform. Previous roles include Director of Data Science at GraphLab/Dato/Turi, machine learning researcher at Microsoft Research, Redmond, and postdoctoral fellow at Carnegie Mellon University. She received a Ph.D. in Electrical Engineering and Computer science, and B.A. degrees in Computer Science in Mathematics, all from U.C. Berkeley.

Top DSC Resources

• Article: What is Data Science? 24 Fundamental Articles Answering This Question
• Article: Hitchhiker’s Guide to Data Science, Machine Learning, R, Python
• Tutorial: Data Science Cheat Sheet
• Tutorial: How to Become a Data Scientist – On Your Own
• Categories: Data Science – Machine Learning – AI – IoT – Deep Learning
• Tools: Hadoop – DataViZ – Python – R – SQL – Excel
• Techniques: Clustering – Regression – SVM – Neural Nets – Ensembles – Decision Trees
• Links: Cheat Sheets – Books – Events – Webinars – Tutorials – Training – News – Jobs
• Links: Announcements – Salary Surveys – Data Sets – Certification – RSS Feeds – About Us
• Newsletter: Sign-up – Past Editions – Members-Only Section – Content Search – For Bloggers
• DSC on: Ning – Twitter – LinkedIn – Facebook – GooglePlus

Follow us on Twitter: @DataScienceCtrl | @AnalyticBridge