Do we really have a data obesity problem?

The constant search for something bigger might be part of the American culture. However, big data is often critical: without real time credit card fraud detection - a big data application - no store would accept credit cards.

There has been a few people questioning the value of big data recently, and predicting that big data is going to get smaller in the future. While most of these would-be oracles are traditional statisticians working on small data and worried about their career, or practitioners in small countries (Canada and France in particular) who do not have access to big data, I was surprised to see Mike Jordan - a famous machine learning professor at Berkeley - going in the same direction. Obviously big data will be dead in 5 years: it will be replaced by titanic data.

The purpose of this article is to discuss whether or not data grew too big, to the point that it is better (for some companies) to not harvest big data and reverse back to a small-data world, or maybe even a data-free world, where executive decisions are made based on gut feelings.

Here I do not criticize the gut feeling approach, I actually use it a lot myself with great success, as a growth hacker, though it needs tremendous vision to get it to work. I just want to share some thoughts:

  • Big data has tremendous potential, including as an investment device
  • Big data is necessary for more and more companies in our highly competitive environment
  • Big data can be cheaply accessed and processed using vendors, rather than (or in addition to) home-made solutions
  • Intuition, gut feelings from a visionary data scientist, combined with big data, is the way to go
  • Collecting the right data, using the right KPI's, is critical

Part of the problem with big data is cultural. Americans want bigger stuff:

  • bigger cars, burning tons of gas and difficult to park in any city
  • bigger healthcare, costing more and delivering less
  • bigger stores, selling bad food
  • bigger breasts, to the point it is totally un-attractive
  • bigger houses with expensive mortgage
  • bigger universities teaching outdated material and charging exorbitant fees with no guarantee of positive return
  • bigger hamburgers that will send you straight to the hospital

But sometimes, bigger is better. A bigger army, if used properly, will yield significant benefits (unless it is too big and costs too much taxpayer money). And armies require big data to work properly - think about the NSA. Bigger universities, if well managed, are good: it allows each student to choose courses from a very large pool of professors - including highly specialized training that small universities can't afford to deliver.

Finally, big data can be a great investment for any company. Start collecting data now even if you don't use it: when you sell your company, your data might be one of your core assets. Besides, big data is cheap and easy. However you need to carefully chose the metrics and data that you want to invest in, and harvest. Just like any investment after all.

In our case, as a small company (zero employee, 7-digit yearly profit), we leverage big data from our vendors. The value is tremendous. It is used in our growth hacking strategy, and as an unfair competitive advantage. As an example, we use smart computational advertising

  • to operate the largest and fastest growing Twitter profile among all data scientists,
  • to efficiently advertise on AdWords using thousands of smartly selected keywords (updated regularly),
  • to discover how our competitors are growing their traffic
  • to optimize our content, as a digital publisher

Related Article: Definition of big data

DSC Resources

Additional Reading

Follow us on Twitter: @DataScienceCtrl | @AnalyticBridge

Views: 942


You need to be a member of Data Science Central to add comments!

Join Data Science Central

© 2021   TechTarget, Inc.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service