Data scientist is the ‘Sexiest Job of the 21st Century’, so say Thomas Davenport and DJ Patil in their seminal 2012 Harvard Business Review article.
Since then, data scientists have been very much in demand and their wages have soared. Compared to statisticians, computer programmers and AI guys (who are still regarded as geeks), they are extremely well paid.
But are they worth it? Is there a good reason why data scientists should get paid more or are their pay packets over-inflated due to unnecessary hype?
According to academic research by Anders Ericsson and popularised by Malcolm Gladwell in his book ‘Outliers: The Story of Success’, that’s how long it takes to become an expert in something.
Well actually it’s a bit too general. 10,000 hours is an average across lots of disciplines including sport, business, education, music, etc.. Obviously it’s not going to take you 10,000 hours to be an expert in tiddlywinks, snap or noughts and crosses, but if you want to be an expert in 3D chess, quantum physics or neuroscience, then it probably will.
Ericsson also points out that the quality of the practice time is important too. Reading general anatomy books for 10,000 hours will not make you an expert in open heart surgery, but making specific improvements to your knowledge, technique and experience on a daily basis will.
So let’s say – for the sake of argument – that 10,000 hours is what it takes to be an expert in any serious academic pursuit.
In a typical work year of 2,000 hours, then, it’ll take you around 10 years to become an expert.
Yes, my maths is correct(ish) – I’m factoring in how much time you spend on emails, ebay, funny cat videos on TwitFace and the worthless one-hour meetings that interrupt at regular intervals.
So there you have it – dedicate yourself to your chosen discipline and you’ll be an expert in around 10 years.
Well OK, it might take 10 years to become an expert, but it’ll take much less than that to become pretty good at something. But how long?
This is where the Pareto Principle comes in handy.
According to the Pareto Principle, also known as ‘The 80/20 Rule’, you can get 80% of the required results by putting in 20% of the effort, so according to Pareto you can get 80% of the way towards being an expert in 20% of the time. In terms of Ericsson’s 10,000 hours, that’s 2,000 hours or 2 years.
That’s pretty good, really. It means that you only need to work hard for 2 years to be a nearly-expert.
When I look back over my career I sort of recognise this. After a couple of years I’m beginning to get pretty good at something, after which, improvements come much more slowly – according to the flip-side of Pareto’s coin, the final 20% of the way to becoming an expert takes 80% of the effort.
But what does this really mean? If it only takes 20% of the time to attain 80% of the knowledge, then surely the stuff that defines an expert must lie in the remaining 20% that you haven’t reached.
So working for the first 20% of the time gets you strong foundations, but doesn’t qualify you as an expert.
Making good decisions based on your knowledge and experience requires a strong foundation, but making consistently better, more informed decisions requires more than this, it requires expert knowledge and an instinct about your problem domain.
To illustrate this, consider the case – featured in Gladwell’s book – of a lieutenant firefighter who claimed that ESP saved his life and those of his crew. The firefighter’s team was fighting what appeared to be a simple kitchen fire in a one-story house. The lieutenant led his crew into the back of the building to spray water onto the fire, but the fire just kept roaring back. He thought the water should have had more of an impact so he ordered his men back to the living room to regroup. Then, feeling that something wasn’t right, he ordered his men to evacuate the building. As soon as they left, the floor they were standing on collapsed. Had they still been inside, they would have plunged into the fire below.
At the time he considered that he had developed some kind of ‘sixth sense’ that had saved his life, and he claimed that all skilled commanders had it.
He could not have known that the one-story building had a basement, nor did he have any reason to doubt that the source of the fire was in the kitchen. After detailed questioning into the lieutenant’s thought processes, researchers discovered that the firefighter, after many years of training and experience, had developed an instinct about firefighting. The fact that the fire kept roaring back did not make sense for a small kitchen fire; the noise level in the kitchen was too low – fires are usually much noisier; and the living room itself was too hot.
In retrospect, all these indicators made sense for a basement fire below the living room – you can’t douse a living room fire if the fire is coming from the basement; the floor muffled the noise of the fire in the basement; and the heat in the living room was abnormally high because the heat was coming from below.
The lieutenant didn’t have ESP, but what he did have was an instinct borne out of a wealth of experience.
This type of instinct is what, in my opinion, demarcates the boundary between expert and non-expert.
We’ve established what it takes to become an expert in a chosen field, how much dedication it takes to build firm foundations, and we’ve looked at the domain-specific instinct that differentiates between an expert and a non-expert; the ability to know when something is not right even when you can’t quite put your finger on why.
So far, so good.
Let us now apply these to data science.
Data science, unlike many other academic pursuits, is not a single field – it is a diverse and complex multi-discipline of mathematics, statistics, computer programming, artificial intelligence, high performance computing, data visualisation, image analysis, domain-specific knowledge, innovation and much, much, MUCH more. The definition is so broad that it is not possible for any individual to be regarded as an expert in each area. Typically, a data scientist might be an expert in 2 or 3 different sub-sections of data science, such as programming and stats, whilst having a good foundation in a number of other areas, like image analysis, artificial intelligence and data visualisation.
So how long will it take to legitimately be known as an expert data scientist? That all depends on how long it’s taken you to build instinct in each sub-section.
For me, I began to notice, listen to and trust my instinct in medical statistics at around the 5 year mark, and similarly in mathematics, artificial intelligence and in handling medical data. I confess that I haven’t yet reached this point in computer programming, data visualisation, big data techniques or in image analysis, nor in domain-specific areas such as physics, biology and medicine, despite logging more than 3,000 hours in each.
Although I have a cumulative experience of around 30 years in areas in which I have gained an instinct, and another 20+ cumulative years in areas in which I haven’t yet attained an instinct, I don’t really regard myself as an expert in data science. The more I learn, the more I realise I don’t know, and data science is so broad a subject that I doubt I’ll ever consider myself an expert.
I guess that’s OK. I continue to learn new things every day, and I’m not really interested in being labelled one way or another. Being known as an expert (or not) doesn’t change the person that I am or the skills, knowledge, expertise and instincts that I have, but I digress.
What it takes to be known as an expert in something is to have gained sufficient experience to have developed an instinct about it. For data science, an expert should have at least 2 or 3 subject area instincts, and preferably 4, 5 or more. If each takes around 5,000 hours – 5 years – to develop, then an expert would need to have been working on data science sub-disciplines for a minimum of 10 to 15 years, and better still, 15 to 25 years.
Statisticians aren’t highly paid. Neither are mathematicians, computer programmers or AI guys. Speaking from personal experience I would argue that they are mostly over-worked, under-paid and under-appreciated.
Expert statisticians have breached the instinct point and reached the other side. They each will have logged several thousand hours to get there. Similarly for mathematicians, computer programmers and AI specialists. Kudos guys – that’s no mean feat.
Data scientists, on the other hand, have to breach the instinct point in multiple of these sub-disciplines on the way to becoming experts, and have to be on the road to gaining multiple additional instincts in other areas.
It’s not that good data scientists have racked up the hours. They have. It’s not that they understand lots of different areas of data, scientific and business study. They do.
Expert data scientists have decades worth of instincts about data in many and varied forms.
This is why good data scientists are worth the big bucks.
What do you think?
Are data scientists worthy of their pay packets or are they just innocent (and happy) victims of fate?
What is your journey through data science, and what instincts have you gained along the way?
Join the debate below and let me know your thoughts...
About the Author
Lee Baker is an award-winning software creator with a passion for turning data into a story.
A proud Yorkshireman, he now lives by the sparkling shores of the East Coast of Scotland. Physicist, statistician and programmer, child of the flower-power psychedelic ‘60s, it’s amazing he turned out so normal!
Turning his back on a promising academic career to do something more satisfying, as the CEO and co-founder of Chi-Squared Innovations he now works double the hours for half the pay and 10 times the stress - but 100 times the fun!
PS - Don't forget to connect with me in Twitter: @eelrekab
Other DSC Articles by the same Author