Earth be still. Big data has lost its luster.
Could it be that analyzing terabytes, exabytes and zettabytes of information won’t make us smarter … or, even worse, could it make us wrong?
We’re beginning to see headlines like “Google and the flu: how big data will help us make gigantic mistakes” in the Guardian, “Eight (No, Nine!) Problems With Big Data” in the New York Times and “Big Data: Are we making a big mistake?” on author and Financial Times columnist’s Tim Harford’s site.
If you believe what you read, then big data isn’t the ticket that we once thought it was.
Or, maybe it still is, say a whole host of others. They’ll likely point out that “big data” is simply resting in Gartner’s “trough of disillusionment” at the moment, because, as with most new technologies, the number of failures outweigh the number of successes early on.
So, if you buy Gartner’s theory, we’ll slowly but surely, learn to do big data better, climb out of the trough and onto the Slope of Enlightenment where it will become more and more embraced by the mainstream.
Big data will present us with tremendous new insights, just not quite yet.
Is the Trough an Illusion?
If the big data naysayers and Gartner’s theory are right, then you’d think that vendors and industry leaders would be experiencing less demand. But that doesn’t seem to be happening.
Could the media be looking at a few isolated cases versus the market as a whole?
We asked managers at Alpine Data Labs, Alteryx, Birst, Cloudera, Datameer, ElasticSearch, GoGrid, Metric Insights and Zettaset whether sales are down and if, based on their interactions with customers, they thought big data was a bust, oversold or stuck in that trough of disillusionment.
Here’s what they told us.
Stefan Groschupf, CEO of Datameer: "I believe we've already made it through the Trough of Disillusionment not only because we're seeing huge demand and adoption, but because we're seeing it from traditionally late-adopting verticals like life sciences, healthcare and manufacturing."
Jim Vogt, CEO of Zettaset: "Contrary to media reports that big data is dead, we have seen a definite upswing in demand for our Hadoop enterprise security solution since the beginning of the year. While 2013 was a time for many organizations to run Hadoop pilot projects and kick the tires, 2014 is shaping up as a year when companies move from pilot into production. This also drives demand, as organizations scale up their deployments and begin to integrate analytics/BI applications and Hadoop together into a working ecosystem. It also doesn’t hurt that we provide enterprise customers with a sophisticated solution to protect their data, as security is becoming a top-of-mind concern for users of Hadoop.”
George K. Mathew, President and COO of Alteryx: "We're seeing a steady growth of firms that want to take advantage of all their data. Hundreds of our customers are getting the value out of traditional and big data sources being blended together to perform advanced analytics — ultimately to drive the next best decision. The Alteryx product roadmap by the user experience of data analysts connecting against these sources, achieving scale in data processing, and unlocking the legacy trapped in SAS or SPSS. We believe that the variety of data sources that are blended and analyzed is becoming more important than the size of data."
Alan Saldich, vice president of Marketing at Cloudera: "The hype-reality gap around big data has narrowed considerably. Most of our enterprise customers spent the last few years investigating Hadoop and related technologies to find out what was possible. Those early investigators were really excited, as were the vendors, but the solutions just weren't ready in many cases to solve broad data management problems, so there was some disillusionment.
"At the same time, the technology has changed, and continues to change very quickly. The early adopters who were excited by the possibilities but ran into practical realities that made Hadoop difficult to deploy are now seeing innovation that overcomes those issues.
"Over the past 12 months, big data products built on Hadoop have changed radically. Across our customer base, the ones who pressed forward are actually deploying pretty amazing systems that have many of the capabilities that were missing a year or two ago. The press and analysts in many cases are behind the curve as compared with the large enterprises who are successfully deploying Hadoop at very large scale, and displacing investment in legacy systems because of the lingering skepticism that built up in 2012-2013."
Bruno Aziza, CMO at Alpine Data Labs: "The disillusionment you are referring is related to the fact that companies have spent millions of dollars on big data infrastructure but are still struggling to gain business value from those investments. According to recent reports, only 4 percent of companies can attribute better decision making to the use of big data.
"That’s not acceptable and they can do better. The companies we see succeed are the ones that have understood that analytics is 'big data’s killer app.'
"I don’t mean that they have armies of analysts building more pretty front-end and visualizations on top of large datasets. I mean that they have understood that strong data science at scale is the only way to gain value from ever-growing data.
"To win here, companies need to drive their attention to the “big data” analytical applications that grounded on reliable and scalable math. Doing so, they will create shortcuts and gain competitive advantage by further embedding this math into their processes and push it to the edge of their organization."
Jen Grant, CMO of ElasticSearch: "We see some customers that have invested in 'big data' and are frustrated with the results. But big data was defined to them as storage solutions that scaled. I think a lot of the frustration in the current hype-cylce sets in because merely storing all of your data does not achieve what a typical business leader wants.
"They want to create a competitive advantage by being smarter than their competition. Capacity is only the first step.
The ability to search, analyze and visualize that data — in real time — and then put it in the hands of people that can take action is the true promise of 'big data.' We have actually seen an acceleration in interest in the ElasticSearch ELK stack because while people are growing a bit tired of the hype around Big Data, they quickly realize extracting value from environments like Hadoop is key.”
Brad Peters, Chairman and Chief Product Officer of Birst: "Although big data was once viewed as the golden child of tech, its bloom is fading in terms of the value that it is able to deliver all on its own.
"Not that long ago, the focus was on finding, capturing and storing data. Today the shift in focus is to unlocking the value from each and every piece of data we can uncover. When we combine 'big data' with the new generation of analytical tools, we move beyond simply capturing mountains of data and instead you can create actionable information that can be used every day to drive business value.
"Big data in and of itself doesn't drive value to your bottom line — it’s what you do with that data that makes the difference.
"Shifting focus from capturing to refining and presenting data in new ways so that companies can then leverage it to maximize productivity, boost revenue, drive sales, etc., is ultimately going to be the driver of competitive advantage and will separate the market leaders from those that can't keep the pace."
Marius Moscovici, founder and CEO of Metric Insights: "It's not a big data slow down per se, rather, the market has hit a transition point in how companies derive value from data.
"We've hit the next stage in the business intelligence maturation cycle. Big data has become in essence a victim of its own success to the point where the everyday executive is completely bombarded with dashboards. This is why there will no doubt be further industry consolidation and the companies that emerge from the pack will have a renewed emphasis on pushing out the data that impacts companies most in the context of what is happening with a business versus focusing on visually appealing dashboards that don't get opened. Welcome to the era of BI 3.0."
Tony Barbagallo, vice president of Product Management and Marketing at ClustrixDB: "There are always early adopters for new technologies, and Big Data Analytics is no exception. From our vantage point as a SQL database vendor we have specifically noticed an initial flock of momentum to the NoSQL movement because it has garnered appeal from developers who like the freedom to develop in their language of choice.
Unfortunately, the very freedom they enjoy is creating undue complexities as they must essentially rewrite the SQL language to accomplish the more complex analytic queries that are needed to realize the information from their data warehouses. From our standpoint, while NoSQL technologies are finding their way in the big data landscape, scale-out SQL technologies like ClustrixDB are actually starting to see an uptick in demand.
In the end, the desire to analyze and act on large amounts of data will never wane, although the technologies required to accomplish those analytics easily and efficiently may change and in the end, I think they will all come back around to SQL.”
John Keagy, CEO of GoGrid: "Trough of disillusionment? We’re actually seeing a huge uptick and interest in Big Data solutions — especially if you can eliminate the steep learning curve and time-consuming task of deploying those solutions through automated orchestration. At GoGrid, the biggest Big Data demand is from our customers in the digital advertising, e-commerce, and healthcare verticals."
Is the Big Data Backlash Real?
Needless to say, it depends who you ask, but from where we sit and from the vendors’ perspective, interest in big data isn’t waning. What is waning is the hype; it’s being replaced by big data analytics and the Enterprise’s insistence that we make big data discoveries that will have impact.