There are some pretty safe bets to make around big data in 2014, so we’re going to make those and leave it to some of the leading big data innovators -- Cloudera, Hortonworks, MapR, Microsoft, Pivotal and SAP go out on the limb. 

Our Predictions

  • By the end of Q3, “big data” will be known simply as “data.”
  • “Hadoop on the Cloud” will be on the table in enterprise strategy discussions. The business will want in on the conversation.
  • There will be acquisitions and consolidation in the Hadoop ecosystem.
  • More workers will be leveraging “big” data and the new breed of analytics to make small, as well as big, decisions.
  • Security becomes as important as big data, cloud, social and mobile

What the Vendors Say

Enough from us, we invited six industry leaders to make up to five predictions for 2014. Here are their answers:

Amr Awadallah, chief technology officer, Cloudera

  • 2014 will be the moment of truth for big data -- Customers are getting fed up with all the buzz around 'big data.' They want to see what the technology can do for them -- right here, right now. Cloudera believes the Enterprise Data Hub will rise to this need by allowing companies to consolidate all their data into one system that can run multiple workloads to address virtually any enterprise data need.

  • Stronger Hadoop security -- Hadoop security took a big step forward in 2013 with the release of the Apache Sentry project. We expect to see further innovations in security in 2014 to enable Hadoop to rival legacy data solutions.

  • Big data moves to the cloud -- In 2014 we expect to see a big shift as heavy, backend data systems and applications begin migrating into the cloud. For the first time, enterprises are now comfortable enough to have their core data (their blood) stored outside the company’s walls. Not all industries will make the jump in 2014 (e.g., financial services, healthcare and other industries with stringent security considerations), but we expect enterprises in retail, telecom, automotive and other verticals to make the switch this year."

Quentin Clark, corporate vice president, Microsoft

  • We will start to address the 'last mile' problem in big data. While the last few years have been about maturing big data tools, I think we will start to get at what I call the last mile problem. We’ve put the infrastructure in place for high-end specialists to get value from big data, but how do we put this in the hands of everyday users and let them ask their own questions and get their own insights? Many will agree that the value of big data is only realized when a business takes action -- and people are a big part of taking action. What you will see in the next year is that tools like Excel will be connected to big data back-ends so people can get the power of big data in the tools they use every day.

  • We will begin to see a large percentage of big data and Hadoop solutions include data that is stored on premises and in the cloud. We have learned that so much of the value of big data comes from the diversity of the data. But it is not always diversity in new data signals. Actionable big data comes from data sources out of traditional business applications, out of new signals and out of external signals like industry partners, government data and even other industries.

  • The industry will begin to shape around data models that will help organizations maximize the value of their data. The questions and insights we can get will deeply join the conversation in the coming year. Composing the right variety of data, at the right scales and on a speed-of-business cadence provides the right environment, but the models and questions we ask are the other important component. To get to the right insights and impact -- to achieve actionable big data -- we have to be asking the right questions over the right perspective on the data. We are at the early stages of the industry shaping around models that help every industry, every company get value from big data / Hadoop, and then start to maximize that value."

Steve Lucas, president, SAP Platform Solutions at SAP

  • Big Data will force a new storage architecture to emerge: As the conversation around big data grows beyond ‘data warehousing’ and towards integrating it into core business processes, there will be a growing recognition that big data is a velocity problem first and volume problem second. This will drive the conversation towards technologies that accelerate the end-to-end process to acquire, analyze and act on big data. Furthermore, an increasing concern about the cost of storing massive data sets, and a continued lack of expert resources with NoSQL technologies like Hadoop, will drive a new emphasis on the cloud friendly solutions. However, this emphasis will also force companies to pay attention to advancing questions around data privacy and security, and will force a new infrastructure / roadmap to emerge that can address all concerns.

  • The cloud will develop beyond storage to become an innovation platform: The days of building a business case to move to the cloud based on total cost of ownership are over. Companies will increasingly look to invest in the cloud as their innovation platform -- as a way in which they can extend their existing infrastructure, enable new processes and achieve new insights that enable them to run their bus in an entirely new way -- not just do the same old thing using a different delivery model.

  • More looking forward, less looking back: Organizations of all shapes and sizes have trumpeted the value of knowing your data and using it for predictive purposes. 2014 is the year that Business Intelligence makes the leap, with enterprises truly embracing and using their historical knowledge to make predictive leaps -- looking forward and employing predictive analytics to 'skate where the puck is headed.'"

John Schroeder, CEO and cofounder, MapR Technologies

  • SQL simultaneously becomes the biggest promise and disappointment for big data. SQL development for Hadoop lets business analysts use their skills and SQL tools of choice for big data projects. Developers can choose from Apache projects Hive and Drill, Impala and proprietary technologies such as Hadapt, HAWQ, and Splice Machine. But SQL requires data structure. Centrally structuring data causes delays and requires manual administration. SQL also limits the type of analysis. An overemphasis on SQL will delay organizations fully leveraging the value of their data and delay reactions. Even in the 1980s, companies could not keep up with the rate of new data source ingest and change when building and administering that centralized structure.

  • The three top Security concerns in 2014: Authentication, Authentication, Authentication. With an onslaught of access control capabilities available in Hadoop, organizations quickly realize that wire level authentication is the required foundation. Without adequate authentication, any upper level control is easily bypassed thwarting intended security initiatives.

  • Emergence of Operational Hadoop. 2014 will see a dramatic increase in production deployments of Hadoop by companies across industries which will reveal the power of Hadoop in operations where production applications combine analytics for measureable business advantage in applications such as customized retail recommendations, fraud detection and leveraging sensor data for prescriptive maintenance.

  • Every industry leader will deploy a new data centric application or they won’t be leading for long. The ability to leverage big data will emerge as the competitive weapon in 2014 as more companies will use big data and Hadoop to pinpoint individual consumers’ preferences for profitable up-sell and cross-sell opportunities, better mitigate risk, and reduce production and overhead costs.

  • Search emerges as the Unstructured Query Language. In 2013 we saw a large number of SQL initiatives for Hadoop. 2014 will be the year that the unstructured query language comes into full focus. Integrated Search into Hadoop provides a simple intuitive method for any business user to locate important information. Search engines are also the core for many discovery and analysis applications including recommendation engines.

Todd Paoletti, VP of Product Marketing, Pivotal Inc.

  • This will be the year of the data lake; where companies can and will provide data to all constituents of the enterprise. In 2014, we’ll also continue to see innovation around making Hadoop more useable with advanced real time and deployment capabilities. We started to see a rise in big data applications, but this will be the year that companies can take their data, apply Hadoop as a source to analyze the queries, layer real time capabilities on top, analyze data from the app and continue to repeat the process to derive new value for the company.

Jim Walker, Director of Product Marketing, Hortonworks

  • Big Data becomes just 'data.' The market will move past the 'big data' moniker and focus on 'data' -- be it big, small, fast or otherwise.

  • Security and data management will quickly emerge as the two most important Hadoop requirements as organizations look to extend use of Hadoop to gain business advantage.

  • Purpose-built, vertically focused solutions will develop, enabling mass adoption of Hadoop.

  • Hadoop will become more transactional; the utility of Hadoop will continue to get broader.

  • Existing software vendors will release YARN-enabled version of existing business applications that take advantage of the power of Hadoop to provide new value to customers."

What’s Your Prediction?

We’ll be following your comments between now and 2014, so go out on a limb and weigh in.