Earth be still. Big data has lost its luster.
Could it be that analyzing terabytes, exabytes and zettabytes of information won’t make us smarter … or, even worse, could it make us wrong?
We’re beginning to see headlines like “Google and the flu: how big data will help us make gigantic mistakes” in the Guardian, “Eight (No, Nine!) Problems With Big Data” in the New York Times and “Big Data: Are we making a big mistake?” on author and Financial Times columnist’s Tim Harford’s site.
If you believe what you read, then big data isn’t the ticket that we once thought it was.
Or, maybe it still is, say a whole host of others. They’ll likely point out that “big data” is simply resting in Gartner’s “trough of disillusionment” at the moment, because, as with most new technologies, the number of failures outweigh the number of successes early on.
So, if you buy Gartner’s theory, we’ll slowly but surely, learn to do big data better, climb out of the trough and onto the Slope of Enlightenment where it will become more and more embraced by the mainstream.
Big data will present us with tremendous new insights, just not quite yet.
Is the Trough an Illusion?
If the big data naysayers and Gartner’s theory are right, then you’d think that vendors and industry leaders would be experiencing less demand. But that doesn’t seem to be happening.
Could the media be looking at a few isolated cases versus the market as a whole?
We asked managers at Alpine Data Labs, Alteryx, Birst, Cloudera, Datameer, ElasticSearch, GoGrid, Metric Insights and Zettaset whether sales are down and if, based on their interactions with customers, they thought big data was a bust, oversold or stuck in that trough of disillusionment.
Here’s what they told us.
Stefan Groschupf, CEO of Datameer: "I believe we've already made it through the Trough of Disillusionment not only because we're seeing huge demand and adoption, but because we're seeing it from traditionally late-adopting verticals like life sciences, healthcare and manufacturing."
Jim Vogt, CEO of Zettaset: "Contrary to media reports that big data is dead, we have seen a definite upswing in demand for our Hadoop enterprise security solution since the beginning of the year. While 2013 was a time for many organizations to run Hadoop pilot projects and kick the tires, 2014 is shaping up as a year when companies move from pilot into production. This also drives demand, as organizations scale up their deployments and begin to integrate analytics/BI applications and Hadoop together into a working ecosystem. It also doesn’t hurt that we provide enterprise customers with a sophisticated solution to protect their data, as security is becoming a top-of-mind concern for users of Hadoop.”
George K. Mathew, President and COO of Alteryx: "We're seeing a steady growth of firms that want to take advantage of all their data. Hundreds of our customers are getting the value out of traditional and big data sources being blended together to perform advanced analytics — ultimately to drive the next best decision. The Alteryx product roadmap by the user experience of data analysts connecting against these sources, achieving scale in data processing, and unlocking the legacy trapped in SAS or SPSS. We believe that the variety of data sources that are blended and analyzed is becoming more important than the size of data."
Alan Saldich, vice president of Marketing at Cloudera: "The hype-reality gap around big data has narrowed considerably. Most of our enterprise customers spent the last few years investigating Hadoop and related technologies to find out what was possible. Those early investigators were really excited, as were the vendors, but the solutions just weren't ready in many cases to solve broad data management problems, so there was some disillusionment.
"At the same time, the technology has changed, and continues to change very quickly. The early adopters who were excited by the possibilities but ran into practical realities that made Hadoop difficult to deploy are now seeing innovation that overcomes those issues.
"Over the past 12 months, big data products built on Hadoop have changed radically. Across our customer base, the ones who pressed forward are actually deploying pretty amazing systems that have many of the capabilities that were missing a year or two ago. The press and analysts in many cases are behind the curve as compared with the large enterprises who are successfully deploying Hadoop at very large scale, and displacing investment in legacy systems because of the lingering skepticism that built up in 2012-2013."
Bruno Aziza, CMO at Alpine Data Labs: "The disillusionment you are referring is related to the fact that companies have spent millions of dollars on big data infrastructure but are still struggling to gain business value from those investments. According to recent reports, only 4 percent of companies can attribute better decision making to the use of big data.
"That’s not acceptable and they can do better. The companies we see succeed are the ones that have understood that analytics is 'big data’s killer app.'