Comment

Comment by Bosco Albuquerque on February 20, 2014 at 3:40pm

1. (a) Should companies embrace big data? (b) Which ones (start-ups, big-companies, tech companies, retail, health care)? (c) And how? (d)Using vendors, outsourcing or by hiring employees? (e) And how do you measure ROI on big data? (f) Should they use redundant data to consolidate KPI's?

I think these are very interesting questions. I'll answer them one at a time.

(a) Companies should embrace all data. Where skills/technology lack, ROI should be examined to determine whether it is worth their while to explore new data frontiers.

(b) All companies should should all data with the same rationale mentioned in (a)

(c) The approach should be one from Business Strategy. Do we have the data to support business decisions? What data do we need to help make better decisions? What are the opportunities for optimizing business strategy in the areas of vendor negotiations, cost savings, profits, employee retention, hiring the best talent, customer acquisition etc.

(d) A mix of internal domain experts, data scientists, and technology infrastructure experts. Where the role goes missing internally, look to hire permanent staff, but hire contractors when needed. Depending on the situation it may be necessary to hire a specialist consulting company.

(e) I've seen ROI being measured quantitatively in terms of how data has driven decisions which has shown a specific dollar figure in terms of cost savings due to vendor negotiations

(f) Additional parameters/indicators may need to be defined. I'd lean towards keeps things separate i.e. more detail the better.

Comment by Khurram on February 14, 2014 at 3:17am

We cannot measure or weigh in terms of unit for big data , an excerpt from a book "Doing Data science" written by Rachel Schutt & Cathy O'Neil

"Constructing a threshold for Big Data such as 1 petabyte is meaningless because it makes it sound absolute. Only when the size becomes a challenge is it worth referring to it as “Big.”So it’s a relative term referring to when the size of the data outstrips the state-of-the-art current computational solutions (in terms of memory, storage, complexity, and processing speed) available to handle it."

As far as concern about sparse data may be costly or not , depends on which problems/trend you are after e.g. Marketing for a single a product A,B or C or combination of all,  finding a Telco bundles charm for all Customers or some specific brackets customer , Customer whether from retailer or should be from wholesale? Depends  what you want to solve. Yes there should be price for big data storage (Cloud computing) , transformation from Cloud to yours designed model and vice versa would require data washing and traversing right data for yours right problem all requires costing.

For the second part I tried my best to contribute , correct me please if I am wrong. You have got very good stuff in yours question and contribution to these question will gear ones skill to become a Data Scientist.

Comment by Big Data Queen on February 10, 2014 at 1:35pm

Vincent great questions when considering Big Data. When considering a big data strategy, I think it's worth mentioning HPCC Systems from LexisNexis. Designed by data scientists, HPCC Systems is an open source data-intensive supercomputing platform to process and solve Big Data analytical problems and can help companies derive actionable insights from their data.

HPCC Systems provides proven solutions to handle what are now called Big Data problems, and have been doing so for more than a decade. The main advantages over other alternatives are the real-time delivery of data queries and the extremely powerful ECL language programming model. More info at http://hpccsystems.com

Comment by Alex J. Caffarini on February 10, 2014 at 7:48am

To your first question about whether companies should embrace big data, it really depends. What is the company trying to accomplish? All the data in the world is useless if companies don't know what they want to do with it. A company's adoption of Big Data should evolve. It should start with a small project and then expanded as learnings go on. For example, a restaurant chain might start with a summary of top selling menu items in its stores to gauge an idea of ingredients that are popular with patrons. Then they can look at their locations to see if those tastes are regional. Learnings from that can then dictate the use of Big Data for deeper things like selection of vendors, quality control, etc. 

How to use Big Data is another major concern. If a company has no Big Data expertise in-house, it may be difficult to know what it wants to accomplish with it, how to use it, how to hire for it, or to whom they could outsource. In such a case, a company may want to talk to executive recruiters with experience placing Big Data and Data Science professionals. They can alert these companies as to the types of skill sets that are available. Then the company may want to talk to a consultant for the sole purpose of problem definition. Another consideration for outsourcing or in-house - security of the data. If the data is sensitive, keep it in-house. Also, if the local talent pool is replete with experienced data science professionals (as it is in most major metropolitan areas), keeping it in-house might be best. If the local talent pool is lacking of these professionals, consider outsourcing or remote workers. Vendors would be best for short-term Big Data projects or for supplementary data (e.g., demographic data, etc.).

ROI on big data should be measured based on what the company wanted to accomplish. If a restaurant chain used Big Data to determine what ingredients were most in demand with customers' needs, then did the actions they took from the learnings result in fewer ingredient out-of-stocks? Increased sales? Fewer sales of less popular items? KPIs should be specific to the Big Data project at hand.