Vertical vs. Horizontal Data Scientists
There are two types of data scientists:
- Vertical data scientists have very deep knowledge in some narrow field. They might be computer scientists very familiar with computational complexity of all sorting algorithms. Or a statistician who knows everything about eigenvalues, singular value decomposition and its numerical stability, and asymptotic convergence of maximum pseudo-likelihood estimators. Or a software engineer with years of experience writing Python code (including graphic libraries) applied to API development and web crawling technology. Or a database guy with strong data modeling, data warehousing, graph databases, Hadoop and NoSQL expertise. Or a predictive modeler expert in Bayesian networks, SAS and SVM.
- Horizontal data scientists are a blend of business analysts, statisticians, computer scientists and domain experts. They combine vision with technical knowledge. They might not be expert in eigenvalues, generalized linear models and other semi-obsolete statistical techniques, but they know about more modern, data-driven techniques applicable to unstructured, streaming, and big data, such as (for example) the very simple and applied Analyticbridge theorem to build confidence intervals. They can design robust, efficient, simple, replicable and scalable code and algorithms.
DJ Patil, an Horizontal Data Scientist
Horizontal data scientists also come with the following features:
- They have some familiarity with six sigma concepts, even if they don't know the word. In essence, speed is more important than perfection, for these analytic practitioners.
- They have experience in producing success stories out of large, complicated, messy data sets - including in measuring the success.
- Experience in identifying the real problem to be solved, the data sets (external and internal) they need, the data base structures they need, the metrics they need, rather than being passive consumers of data sets produced or gathered by third parties lacking the skills to collect / create the right data.
- They know rules of thumb and pitfalls to avoid, more than theoretical concepts. However they have a bit more than just basic knowledge of computational complexity, good sampling and design of experiment, robust statistics and cross-validation, modern data base design and programming languages (R, scripting languages, Map Reduce concepts, SQL)
- Advanced Excel and visualization skills.
- They can help produce useful dashboards (the ones that people really use on a daily basis to make decisions) or alternate tools to communicate insights found in data (orally, by email or automatically - and sometimes in real time machine-to-machine mode).
- They think outside the box. For instance, when they create a recommendation engine, they know that it will be gamed by spammers and competing users, thus they put an efficient mechanism in place to detect fake reviews.
- They are innovators who create truly useful stuff. Ironically, this can scare away potential employers, who, despite claims to the contrary and for obvious reasons, prefer the good soldier to the disruptive creator.
In my opinion, vertical data scientists are fake data scientists. They are the by-product of our rigid University system which trains people to become either a computer scientist, a statistician, an operations research or a MBA guy - but not all the four at the same time. This is one of the reasons why we have created our data science program. This is also one of the reasons why recruiters can't find data scientists: they find and recruit mostly vertical data scientists. Companies are not yet used to identifying horizontal data scientists - the true money makers and ROI generators among analytic professionals. The reasons are two-fold:
- Untrained recruiters quickly notice that horizontal data scientists lack some of the traditional knowledge that a true computer scientist, or statistician, or MBA must have - eliminating horizontal data scientists from the pool of applicants. You need a recruiter familiar both with software engineering, business analysts, statisticians and computer scientists, and able to identify qualities not summarized by typical resume keywords, and identify which (lack of) skills are critical from the ones that can be overlooked, to detect these pure gems.
- Horizontal data scientists, faced with the prospects a few job opportunities, and having the real knowledge to generate significant ROI, end up creating their own start-up, working independently, sometimes competing directly against the very companies that are in need of real (supposedly rare) data scientists. After having failed more than once getting a job interview with Microsoft, eBay, Amazon or Google, they never apply again, further reducing the pool of qualified talent.
Hopefully, our data science program will help with this - in particular educating recruiters and hiring managers as well.
Question: Can you name a few horizontal data scientists? Vertical data scientists are a dime a dozen.
Related articles:
- Fake data science
- 27 types of data scientists
- How much does a data scientist make at Facebook?
- 66 job interview questions for data scientists
- Why can't Microsoft find analytic talent
- $33K for an outdated applied math degree
- Debunking the lack of analytic talent
- Data Science Apprenticeship
- Data Science eBook
- AnalyticTalent
- Top metalinks from DSC
- Top big data influencers
Comment
-
Comment by Wilco van Ginkel on April 1, 2013 at 8:28am -
Vincent,
Thanks for sharing this article - an interesting read, indeed.
However, it reads like the field of data scientists is a binary field: good (horizontal) or fake (vertical). Which - IMHO - is not the case. I believe that it is (or should be) more about data science teams, where each member adds value & different skills.
In such a team there is a place for both horizontal and vertical data scientists. Depending on the task at hand & scope, the distribution of horizontal and vertical within the team might differ.
Keep up the good writing!
-
Comment by Larry R Myers on March 27, 2013 at 6:52am -
Vincent is right. Our universities are producing specialists in narrow specialties. Elementary education majors are not required to take any mathematics courses during their certification studies, for example.
-
Comment by Vincent Granville on March 20, 2013 at 8:38am -
To clarify:
By "horizontal", I mean broad spectrum as acquired after 10+ years of experience working in various industries with different roles (digital analyst, market research, software engineer, statistician - in finance, advertising and environmental statistics, large companies and start-up founder), combined with deep expertise in a few (usually more than one) domains (e,g. support vector machines, API development, Bayesian networks, Python, big data).
-
Comment by Vincent Granville on March 19, 2013 at 7:07am
-
@Dmitriy: Why not hire an horizontal data scientist with deep domain and technical expertise in advertising, someone just like me actually?
-
Comment by Dmitriy Kruglyak on March 18, 2013 at 9:32pm -
@Vincent:
I beg to differ. If my business is advertising, I want people with some experience in advertising. I am not as interested in people who never even thought about relevant industry problems and only worked on for example designing OCR systems or search engines or health claim prediction or something else that does not readily translate.
"Horizontal" skills are a great building block but they come with a significant cost of ramp-up time to *really* understand the actual problem, the data, the exceptions, the use cases and the business imperatives. As far as giving senior management roles to people without understanding of the industry problems - that is a truly scary thought.
-
Comment by Vincent Granville on March 18, 2013 at 8:20pm
-
@Dmitriy: Vertical knowledge and roles create silos that do not communicate and result in loss of performance. To eliminate silos, you need horizontal managers. It's OK to have vertical scientists at the bottom of the corporate pyramid, but you really need horizontal people higher up - or bring them as management consultants.
-
Comment by Dmitriy Kruglyak on March 18, 2013 at 7:02pm
-
@Vincent: In that case I'd rather call them "multi-vertical". The term "horizontal" usually implies lack of depth in any specific vertical. I've generally seen challenges getting useful results out of pure generalists, they cannot be productive until they spend quite a bit of time getting up to speed on the problem's subject matter (not that it is not viable).
-
Comment by Vincent Granville on March 18, 2013 at 6:46pm
-
@Dmitriy: Horizontal data scientists are indeed domain experts, many times in multiple fields (e.g. in telecom, online advertising, banking industry, clinical trials, fraud, retail) - read my blog. It makes them recession-resistant and change-proof, unlike his vertical colleague.
-
Comment by Dmitriy Kruglyak on March 18, 2013 at 5:39pm
-
This is a bogus dichotomy.
The only thing that matters is whether or not a "data scientist" is able to meaningfully contribute to the bottom line of their employer.
The so-called "vertical data scientists" usually have very specific, defined and useful skills that can make a difference right away - within certain scope / limits.
The so-called "horizontal data scientists" often fail to go much further than producing brainstorms of doubtful value. To accomplish anything useful they have to come down from their high horse, roll up their sleeves and become domain experts in some specific problem - essentially turning into a "vertical data scientist".
When you know the problem you want to solve the choice between mile wide / inch deep and inch wide / mile deep practitioner is pretty obvious.
-
Comment by Vincent Granville on March 18, 2013 at 9:01am
-
@Stephen: Good point. In my case, and as an horizontal data scientist, my salary is zero. But in terms of revenue from running a profitable data science company, I'm far above the upper limits posted in my article on Facebook data scientist salaries (although I admit that these numbers look a bit low). In addition, I live in a state with no income tax and have quite well optimized my finances: be it from a tax point of view, by buying / selling real estate at the right time / right place, optimization of my health, retirement and education expenditures, optimization with respect to the stock market, selling patents, purchasing assets (car, house) far less expensive than I can afford (many times when they are cheap) and making them last longer than most people, etc. All of this thanks to having a strong analytic mindset, that I leverage as much as I can in many aspects of life.
Videos
Top Content
Follow us
© 2013 Data Science Central
You need to be a member of Data Science Central to add comments!
Join Data Science Central