If you are a start-up director in 2016, not a week goes by without someone talking to you about unicorns.
It is difficult to imagine the feeling of riding a unicorn. Do you feel the wind of success rush against its skin? Do you fear the arrows and traps of pitiless hunters? Do you feel a special sense of excitement when you get to the end of the rainbow of profits?
I find it easier to imagine the state of mind of the directors of Cloudera, Hortonworks or MapR, who are now the leading Big Data unicorns.
Three years ago, Wikibon talked about Hadoop to describe the forthcoming war between Hortonworks and Cloudera. Their particular challenge then was to establish themselves as one of the leaders in Hadoop distribution. Their strategic objective then was to keep the traditional software and database players (IBM, Microsoft, Oracle and HP) outside this ecosystem.
Today, in 2016, there are few reliable statistics available on the deployment rates of Hadoop distributions. A Dezire article mentions 53% for Cloudera, 16% for Hortonworks and 11% for MapR. In any case, it seems clear that Cloudera, Hortonworks and MapR share 80% of the market between them. They have therefore bravely succeeded.
But now what is next?
It is not necessarily them to battling it out for a few extra percent. Taking a few extra market points away from Hortonworks is not going to make Cloudera the next Oracle!
In the end, the formula is simple:
The fear of Big Data cracking
Underlying the Hadoop distributions war, a diffuse fear is beginning to take hold of the market, like a fine crack. What if Big Data on Hadoop weren't to work that well?
Indeed, many companies have invested heavily in Hadoop clusters. Most of the technological companies created since 2010 have put their money on Hadoop and built part of their information systems around it. For them, Hadoop is natural. For others, their enthusiastic investment runs the risk of being transformed into the fear of creating "yet another data warehouse", a storage silo the added value of which they will find difficult to justify
The global risk is to see Hadoop reduced to the role of "backup" for large volumes of data (as it is cheaper) or of "sandbox" for certain more adventurous equipment.
Manifesto for demanding Big Data analytics applications
How can we ensure that Hadoop is used more and more for critical added value applications?
The response already does not belong to one sole vendor or to one sole typology of application - probably a large part of the response is found in the implementation of critical transactional applications on Hadoop.
From the point of view of the "analytics" applications (which seeks to analyse data in order to bring an incremental value), certain bias could allow a systematic movement in the right direction:
We could call this corpus of principles a manifesto for demanding and added value applications on Hadoop.
Holding to this is not clear:
Hadoop launched the idea that "global" Big Data platforms were possible. Of course, everything still needs to be created in terms of organisation, tools, practices so that everyone can benefit from them. The preceding revolution in this was the birth of relational databases; and nearly 15 years were needed for it to find their full potential, for example in the democratisation of website creation. We are only at the start!
Cloudera, Hortonworks, etc... will do all they can to stop the Big Data from cracking. But let's all play a part !