Survey: 'Big Data Paralysis' Is Holding Companies Back

Summary:  Recent surveys suggest that Big Data and Hadoop may be stalled.  Why?

Just a week after a report from research firm Gartner Inc. found that investment in Hadoop-based Big Data technologies "remains tentative" with poor adoption rates, a survey from Software AG identified "a Big Data paralysis" that's keeping enterprises from realizing the promised benefits of the analytics craze.

This finding might come as a surprise to those of us in the Big Web User World of Data Science.  That’s a designation I’ve started using to define B2C companies that interact with their customers primarily through the internet and they cover the waterfront from search, to ecommerce, social media, dating sites, MOOGs, and content sites.  Think Google, Amazon, eHarmony, Twitter, Yahoo, and World of Warcraft.  These companies rely on Hadoop and NoSQL databases along with the latest machine learning tech.  Rely isn’t really strong enough.  These companies ARE Big Data and if you’re in San Francisco, Los Angeles, New York, or any of the other start-up havens you wouldn’t think anything is slowing the Big Data movement.

However, the rest of the world, the Core Data Science world is a little more cautious.  This article by David Ramel of 1105 Media in May draws some other interesting conclusions based on these two competing survey:

"While decision makers understand that greater visibility over internal processes and real-time data analysis could lead to operational improvements, they still lack the support and solutions needed to make it a reality," Software AG said last week in a news release about its new study.

"Despite substantial hype and reported successes for early adopters, over half of respondents (54 percent) report no plans to invest at this time," the Gartner report states. "Additionally, only 18 percent have plans to invest in Hadoop over the next two years."

In its own survey, Software AG commissioned Vanson Bource to poll big companies to determine "How Effective Use of Data Improves Operational Processes."

'What Challenges Does Your Organization Experience Around Collecting and Analyzing Data?' (source: Software AG)

But unlike the Gartner report, the oft-cited Big Data skills shortage is not to blame in the Software AG version of Big Data truth. Rather than lack of skills, the big obstacles identified in the Software AG report focus on IT operations, as one of the stated goals of the survey was to "evaluate if organizations are monitoring operational processes and how these processes are prioritized."

Other survey highlights listed by Software AG include:

  • 74 percent of respondents agree that more visibility over internal processes could lead to operational improvements.
  • 65 percent say that it will also decrease the chances of falling behind their competitors.
  • Organizations are most likely to be using quantitative data over other sources to prioritize processes.
  • Around three-quarters (73 percent) agree that analyzing a lot of data is a challenge for their organization.
  • Acting on data in real-time is seen as a challenge for 65 percent.
  • Over eight in 10 (83 percent) state that their operational processes take too long.
  • The majority of respondents' organizations cannot act on (57 percent), mine (61 percent) or use (68 percent) real-time data.
  • 56 percent of respondents' organizations are using manual processes to collect and analyze data.

SiliconANGLE just yesterday followed up on the Wikibon rebutal in an article titled, "Wikibon Analyst Says Hadoop Pessimism is Part of Natural Adoption Cycle."

"Gartner's rather somber outlook doesn't mean that Hadoop has failed to live up to the hype," the article states. "On the contrary, the figures are simply a reflection of what it takes for technologies as complex as Hadoop to mature to a point where they can be easily accepted by mainstream users, argues Wikibon's Big Data and analytics analyst George Gilbert."

Is Big Data and Hadoop stalled?  What do you think?  Read David Ramel’s full article at https://adtmag.com/articles/2015/05/26/software-ag-big-data-study.aspx.



July 14, 2015

About the author:  Bill Vorhies is President & Chief Data Scientist at Data-Magnum and has practiced as a data scientist and commercial predictive modeler since 2001.  Bill is also a Senior Contributing Editor for Data Science Central.  He can be reached at:

[email protected]

Views: 1585

Tags: Big Data, hadoop


You need to be a member of Data Science Central to add comments!

Join Data Science Central

© 2021   TechTarget, Inc.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service