Summary: There is a common misconception that data is the enemy of intuition.  It’s true that sometimes our clients are misled by their preconceived notions.  But all the same, there are two practical lessons here, one about being effective in explaining data science to our clients, and another about methodology. 


There is a common misconception that data is the enemy of intuition.  How many times have we made jokes about the ‘dumb’ actions companies make when they make decisions guided by HIPPO (the highest paid person’s opinion).  Let’s face it, bad decisions make good stories.  All the same, I think there are two practical lessons here, one about being effective in explaining data science to our clients, and another about methodology.

Lesson 1:  Explaining the Data and Dealing with Clients:

The first lesson is about how we ‘tell the story’ of the data to our client.  It’s the old ‘attract more bees with honey’ metaphor applied to client relations.  This is particularly well illustrated by an interview in this morning’s WSJ with Hilary Mason, chief executive and founder of Fast Forward Labs and former chief scientist at Bitly.  She describes this interchange with her client, the CMO of a frozen breakfast sausage brand when she suggested that they find out who his customers really are.  Here’s the interchange.

He said, “I know who my customers are. They’re moms in the Midwest.” I said, “How do you know?” He looked at me like I was crazy. He said, “They’re my customers. I’ve been doing this for years.”

He was not wrong. He had many customers. We didn’t know if they were moms, but they were looking at mom-type things. They were in the Midwest. But he missed a cluster of customers in Texas, and they were into motorcycles and man things. He missed a cluster in the Northwest who were anti food additives [who liked his product because it] did not have these additives.

So her takeaway and the moral of Lesson 1 is that his intuition wasn’t wrong, but he was missing things that were too small to be detected by the radar of his experience, yet important in identifying new markets.

Coming back to the title of this article, ‘Data is Not the Enemy of Intuition’ we will be well guided to tell our experience-burdened clients that data is in fact a tool for enhancing intuition.

Lesson 2:  Don’t Forget to Segment and Cluster

Back to human behavior modeling 101, one of the first things we learn is that our response models can (almost) always be made better by first segmenting and clustering the target buyers and constructing separate models for each cluster.

If Ms. Mason had skipped this step she and her client would not have discovered two new market segments for the product.  Segmenting the population would then result in three different response models probably resulting in each being more accurate and valuable. 

Even after developing the models if they proved to be very similar in features, weights, and accuracy, then that would have been an important piece of information in formulating marketing plans and budgets.

The reason that I raise what seems to be a basic practice has to do with the rise of analytic platforms that increasingly seek to bypass the data scientist and go directly to the well intentioned and motivated manager or analyst we sometimes call the ‘citizen data scientist’.  I was beta testing one of these new platforms recently and the focus was entirely on developing predictive models with the least effort.  Preprocessing and missing data imputes were full automated as was producing the models.

It struck me that there was no guidance or option that even suggested to the naïve user that segmenting and clustering is a logical first step, no matter how accurate the resulting single model may appear.  A single model produced without at least exploring segmentation will always be ‘just good enough’.  Fast is one thing, but leaving profit on the table because you didn’t know better is another.


About the author:  Bill Vorhies is Editorial Director for Data Science Central. and has practiced as a data scientist and commercial predictive modeler since 2001.  He can be reached at:

[email protected]


Views: 1617

Tags: cluster, intuition, segment


You need to be a member of Data Science Central to add comments!

Join Data Science Central

Comment by Justin Veenstra on February 12, 2016 at 12:33pm
If I might suggest: insight from data is not the enemy of intuition. Data are data whether you look at them or not. Some insights are intuitive, and some are not... depending on how you look at them.
Comment by John Byrnes on February 11, 2016 at 10:23am

Nicely stated.

Indeed facts help us in making better decisions, but there is a continuum between facts and intuition when we move from the role of facts to the role of judgement.  I wrote a short piece on LinkedIn about this.  https://www.linkedin.com/pulse/baseball-banking-john-byrnes?trk=pro...

© 2021   TechTarget, Inc.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service