If you’ve followed the big data hype long enough, you know that every year -- since 2011 -- was supposed to be big data’s big year. And that at the end of every year since, the pundits have said “It didn’t happen this year. Next year will be big data’s big year.”
The analysts, of course, have been more cautious than the marketers; in fact, late last year Gartner said that it will take longer (five to 10 years) than they thought for big data to crawl out of its trough of disillusionment and onto the plateau of productivity.
They cite two reasons for this: first, adoption of big data tools and techniques is happening ahead of learned expertise and maturity/ optimization (which if correct is a prescription for disappointment) and second, that enterprises have been unable “to spot big data opportunities” and “formulate the right questions and execute on the insights."
Tom Davenport, in his Harvard Business Review webcast, Big Data at Work: Dispelling the Myths, Uncovering the Opportunities, isn’t as pessimistic. He said 2015 may be the year big data crosses the chasm.
While both Gartner and Davenport are great advisors, we thought it might be interesting to see what people who have their feet on the street are saying. GigaOm’s Structure Data Conference in New York City gave us the perfect opportunity to do so.
“Big data adoption is accelerating rapidly,” said Alan Saldich, vice president of marketing at Cloudera. “While a few years ago customers were asking 'What is Hadoop?' and the next year they were asking 'What can I do with it?' they now have business strategies around it.”
And those strategies, mind you, aren’t just about off-loading data from expensive, proprietary hardware and dumping it all into Hadoop where storage and processing is at a fraction of the cost, but about discovering and applying insights that weren’t conceivable in the past.
Hortonworks is Bullish on Hadoop
Rob Bearden, CEO of Hortonworks is bullish as well. Standing before a packed house at the conference he told the audience that the Enterprise Open Source Hadoop provider would be a $1 billion company in the next four years.
We asked David McJannet, the company’s vice president of marketing if 2014 would be big data’s big year.
We’ve seen significant growth in all the leading indicators of our business -- from number of customers to rate of adoption -- that make it clear that interest in Hadoop has moved beyond investigation. The early majority has moved into real production deployments for their big data projects, and as a result, the big data market is accelerating at an unprecedented rate.”
Hadoop Use is Growing Among MapR Customers
Jack Norris, chief marketing officer at MapR offers a similar take, “We're seeing Hadoop adoption rapidly accelerate as customers are realizing revenue gains, risk mitigation and cost reduction. Customers are also expanding their use within their companies. More than 80 percent of our customers expand their Hadoop cluster within the first year.”
Pivotal Helps Customers Make Important, Strategic Pivots
There was no disillusionment at a jam-packed presentation by Christopher Stevens, field director of data science at EMC spinoff Pivotal; the room was so packed that more than a dozen people sat on the floor, never mind all who were standing. Stevens cited examples of how data from both public and private sources could be brought together and analyzed to do important things like alert a family that conditions were ripe for activating their son’s asthma condition and that his supply of medication was about to run out.
Demand for Alpine Data Labs Fueled Insight is Growing
Bruno Aziza of Alpine Data Labs hosted a panel during which Michael Cavaretta, chief data scientist at Ford, Peter Memon, director data analytics at Barclays Capital, and Vikram Vatsalan, executive director at Morgan Stanley, each pointed to what they were doing with big data, but also talked about business wins and lessons learned, proof positive that big data was thriving where they worked.
Steven Hillion, founder of Alpine Data Labs, says that now that Hadoop is maturing as a platform and the foundation is laid for scalable data science, he sees 2014 as being the year that big data begins to provide actionable analytics value for the business.
It's going to break away from the realm of science projects, and start producing valuable insights and analytics that are actually operational,” says Hillion. “An example that we've seen at Alpine is where we've been able to automate the process of training models and pushing them to a website for optimizing end-user experience and increasing conversions.”
And Ankur Gupta, general manager at Sears’ big data spinoff MetaScale, says that now that big data technologies have matured, use cases have been proven and many of the open source related suspicions have been taken off the table, they are seeing an increasingly large number of companies moving from "I want to understand what Big Data is" to "let's talk about a POC and establish success criteria." As a result, they’re hiring to meet this growing demand.
Momentum to Cross the Chasm
And while we couldn’t interview everyone at the conference, we will say this -- we didn’t see disillusionment or even mere curiosity as we had in years past. The enterprise users we spoke to were all in on big data and many of them had been working with it for at least one year.