Here is my top 7 list of daft things that some people say about Big Data.
I think that Big Data does play a role in some businesses. I also think that some of the basic distributed file store and text search technologies can be usefully employed, in non-traditional indexing, counting and correlation. However, there is an awful lot of nonsense said about Big Data.
So, onwards and upwards.
If Big Data is currency, and for most of us, it isn't, then it's more like the hyperinflationary money of the Weimar Republic, rather than something you would take to the bank or try and buy the weekly grocery with.
Big Data might have value, no doubt some of it does – it can't all be dross, can it?But, that doesn't make it a solid financial asset class – that's just dopey. The value of data providers, such as those companies supplying financial market and instruments data, is in the service of providing accurate, appropriate and timely data. Data has no significantly greater intrinsic financial value or exchange liquidity than pints of beer or glasses of wine. It can have time and place utility, of course, as a product or as a service, but it is not like a central-bank backed currency – not even close.
Call me an old-fashioned cynic, if you must, but I don't believe for one moment that we have now learned how to turn lead into gold, or for that matter, Big Data into golden nuggets.
You see, if Big Data contains gold, and it doesn't, then it would be more like FeS2 than anything else. Or, to use the vernacular, it would be more like fool's gold than Welsh gold. Which, considering the quantity of hype surrounding Big Data, is the most apt analogy.
If you want to see what is important for everyone, then start with Maslow's hierarchy of needs, a pyramid of essential human motivators that curiously hasn't been expanded to include Big Data. Maybe because the fact that it's not an essential human need.
I would also like to mention that quite a number of Big Data projects have nothing to do with Big Data or Hadoop at all.
Doing professional taxi-drivers out of a living income is not solving world problems, no matter how many times one chant's Big Data Uber alles.
We know what the fundamental world problems are, and we know, more or less how to solve them. In this respect we don't need Big Data to tell us which way is up. Consider this:
Maybe Big Data can inform us that...
"The silent killers are poverty, hunger, easily preventable diseases and illnesses, and other related causes. Despite the scale of this daily/ongoing catastrophe, it rarely manages to achieve, much less sustain, prime-time, headline coverage." Source: Global Issues
Maybe if we knew all of this (and we really do) then maybe we can do something about tackling the problems.
Maybe we should also be less eager to instrumentalise real suffering in order to flog aging technology and price-gouging services, and actually do something about problems we know of, using solutions that are available to us, to the benefit of those less fortunate than us.
Big Data is characterised by its volumes, its velocities of generation and communication and its varieties.
The thing is, data has always been growing in volumes, and there has never been a time when it has actually decreased. Also, data has been increasingly generated at a faster rate, a trend that doesn't look like stopping anytime soon. Lastly, the varieties in format and content of data objects has been growing since the early eighties (that much I can vouch for), and to some significant extent, before then as well.
The thing is, nothing of significance is new. Not that this matters. But data has always about volumes, types and velocity, and those factors didn't suddenly become relevant at the turn of the millennium. Neither is the technology new, most of the technology labelled as Big Data technology is a collection and configuration of technologies that are decades old.
To be precise, this is about variations on the theme of distributed file systems and text search and count (e.g. Hadoop) replacing Enterprise Data Warehousing.
There are two axioms that can be applied when considering Hadoop and Data Warehousing:
In short, using Hadoop as part of the tech stack for an Analytics Data Store makes absolute sense. Trying to shoehorn a mature strategic and tactical decision support platform into Hadoop is just daft.
Whilst we are on the subject, even staging data with Hadoop is overkill, and simply adds another point of failure in the Data Warehouse process.
As for doing ETL with Hadoop and Java? Please….
There are legitimate and ethical proponents of Big Data and indeed some applications seem, at least superficially, to make some good sense. However, there are an awful lot of unscrupulous or wilfully uninformed Big Data punters around, who are in my view giving the impression that there are no limits to what can be achieved with Big Data. As if Big Data were the ultimate universal panacea.
Of course, to put it simply, it's not true, it never was, nor is it likely that this will ever be the case.
It has been proven that Big Data technology is not only useful for some companies, but it's absolutely essential for them. However, not everyone will have the same or similar business models and drivers as Google, Twitter, Facebook and Amazon. In fact, most other businesses are not really like these internet advertising companies in any significant way when it comes to the data they collect and the ways that they can apply it. That is, very few other companies are basing their business models on advertising revenue streams, data brokerage and search.
That stated, efficient distributed file stores, text search and word counting will have their place, it's just that this place isn't for everyone.
Many thanks for reading
As always, please share your questions, views and criticisms on this piece using the comment box below. I frequently write about strategy, organisational leadership and information technology topics, trends and tendencies. You are more than welcome to keep up with my posts by clicking the ‘Follow’ link and perhaps even send me a LinkedIn invite. Also feel free to connect via Twitter, Facebook and the Cambriano Energy website.
For more on this and other topics, check out my other recent posts:
#Hadoop #BigData #BigDataAnalytics #Decency #Ethics