The constant search for something bigger might be part of the American culture. However, big data is often critical: without real time credit card fraud detection - a big data application - no store would accept credit cards.
There has been a few people questioning the value of big data recently, and predicting that big data is going to get smaller in the future. While most of these would-be oracles are traditional statisticians working on small data and worried about their career, or practitioners in small countries (Canada and France in particular) who do not have access to big data, I was surprised to see Mike Jordan - a famous machine learning professor at Berkeley - going in the same direction. Obviously big data will be dead in 5 years: it will be replaced by titanic data.
The purpose of this article is to discuss whether or not data grew too big, to the point that it is better (for some companies) to not harvest big data and reverse back to a small-data world, or maybe even a data-free world, where executive decisions are made based on gut feelings.
Here I do not criticize the gut feeling approach, I actually use it a lot myself with great success, as a growth hacker, though it needs tremendous vision to get it to work. I just want to share some thoughts:
Part of the problem with big data is cultural. Americans want bigger stuff:
But sometimes, bigger is better. A bigger army, if used properly, will yield significant benefits (unless it is too big and costs too much taxpayer money). And armies require big data to work properly - think about the NSA. Bigger universities, if well managed, are good: it allows each student to choose courses from a very large pool of professors - including highly specialized training that small universities can't afford to deliver.
Finally, big data can be a great investment for any company. Start collecting data now even if you don't use it: when you sell your company, your data might be one of your core assets. Besides, big data is cheap and easy. However you need to carefully chose the metrics and data that you want to invest in, and harvest. Just like any investment after all.
In our case, as a small company (zero employee, 7-digit yearly profit), we leverage big data from our vendors. The value is tremendous. It is used in our growth hacking strategy, and as an unfair competitive advantage. As an example, we use smart computational advertising
Related Article: Definition of big data