Guest blog post by Rubens Zimbres, PhD.
This article brings images from my work modeling with Mathematica, my experience as a Business Analyst and also my doctorate lessons. For me, the borders between a properly executed Business Intelligence and Data Science (with substantive knowledge in Management) are fuzzy.
What is a Data Scientist ? In my understanding, someone can be a data scientist according to his domain expertise: Business management, physics, computer science, etc.
DATA SCIENCE AND BUSINESS INTELLIGENCE PHASES
1) UNDERSTAND PROCESSESFirst of all, really understand the context, processes of the business: familiarity with technology, employees and daily routine
2) FINANCIAL ANALYSIS
Second, establish business needs (among them, $$$).
- Net Worth
- Gross margin
- Net profit
- Indexes: ROI, ROA, ROE, EBITDA, inventory turnover, liquidity, financial leverage, debt, assets and liabilities (short term and long term), horizontal and vertical balance analysis
3) DEFINE DATABASE ARCHITECTURE AND METHODOLOGY OF DATA COLLECTION AND EXTRACTION
Third: a) Define database architecture to provide functionality, reliability, security and ability to provide valuable data for decision making.
b) establish a methodology of data collection, sampling and market research, sources of data and KPIs in order to get a reliable data analysis provided with validity.
4) COLLECT DATA
From different sources:
a) Customized market research
b) CRM Database: sales, clients, suppliers and processes
d) Online Advertising
f) Big Data
- Facial recognition
- Speech recognition
- Unstructured data
- Structured data
- Social Media
5) ANALYZE DATA
You can use Excel, R, SAS, Mathematica, SPSS, Pyhton
5.0. Data preparation: work on missing values, outliers (I usually analyze deeply individuals with values more than 3 standard deviations), normality of data, skewness (the 1/N trick), kurtosis (the log trick), sampling. Prepare data properly so that you can have a reliable analysis.
5.1. Descriptive statistics:
a) Market Research and Database: quality perception, source of clients, demographics, sales, profit, repurchase intentions, profitable clients, profitability per sales channel, losses, evolution of KPIs over time, sales per state/neighborhood, efficiency of employees and sales force, employee performance
b) Social Media: popularity, sentiment analysis, references, associations, conversions, mentions, influencers. You can use Python for unstructured data analysis (text).
c) Website: visits, paths, time spent, clients' demographics, OS, enter pages, leave pages, contact forms filled, popularity, page rank
d) Online advertising: bids, keywords, conversion rate, effective contacts, ROI, clients' demographics, competition strategy
6) DEVELOP SIMULATION MODELS
7) MAKE STRATEGIC DECISIONS TO GET SUSTAINABLE COMPETITIVE ADVANTAGE
8) REPORTING AND GETTING FEEDBACK OF EMPLOYEES, CUSTOMERS AND DATA
P.S. One of the restrictions I have with Big Data is that 80% is unstructured data. Any good academic researcher in Management field knows there is not a well stablished theory in academic literature to proper measure unstructured data, even with content analysis in qualitative research. It would take more than 5 years to have a reliable way and a stablished theory to analyze unstructured data, because academic literature lacks consensus regarding measurement and analysis.
Cognitive bias always exist and it's unavoidable. Even worse if it's an automated algorithm. So, if we take this epistemological critic into account, the foundations of Big Data admiration will be shaken. It will take some time to properly analyze such amount of data. Second, in order to analyze Big Data, one has to be very skilled in analyzing ordinary data, in order to have valuable insights because what really matters is quality, and not volume of data. What matters is not complexity of data, what matters is complexity of the data analyst mind. We can make miracles with small amounts of data, properly analyzed.
Probably the biggest difference between Data Science and Business Intelligence is Machine Learning.