In 2012 we heard, "What is Hadoop?" In 2013 the question evolved into, "What do I do with Hadoop?" Today's question? "What’s the best way to do it with Hadoop?"
I see four emerging trends that will play a major role in the big data and Hadoop space in 2014. Let’s jump right in.
1. Time to Walk the Walk
Companies' big data journeys are on the way. Organizations have bought into the benefits of Hadoop, and understand the value of at least some of the key use cases. As they build their first initiatives, they are looking for tools and best practices to deliver on their promises. In other words, it’s time to walk the walk.
For those of us who have been in this industry long enough, it’s safe to say that along with tons of learning experience, we’ll also enter a phase of disillusionment, where many will have to learn the hard way to separate hype from reality.
2. Back to … the Future?
Not that long ago, in a world not that far away, organizations used hand-coding, primarily SQL, to perform all their data integration (and reporting and analysis, and so on) tasks. Business intelligence and data management solutions promised to bring a new order: graphical user interfaces, metadata and business rules heralded a world of high productivity and reusability without the need for manual coding.
Unfortunately, it didn’t last long. With tons of disparate tools such as HiveQL, Pig, Java and even good ol' C#, organizations are struggling to get productive again.
The good news is, the universe of big data and Hadoop seem to be playing “Back to the Future” all over again. With commercial and open source vendors rushing to fill in the functional and usability gaps, expect lots of progress in 2014 to bring a new order where data scientists are not the only ones allowed into the party.
3. From Silicon Valley, to Wall Street, to Main Street
Hadoop has enjoyed a very happy childhood in Silicon Valley -- loved and pampered by many high-tech companies. Its influence was mostly limited to high-tech companies with the skills and expertise to extract its benefits. But the tide is rapidly changing as more and more traditional businesses start to leverage Hadoop (in part as a result of trend number two above).
Wall Street, telcos and other traditional businesses are realizing they can offload heavy and expensive workloads from legacy systems (read mainframes) to the Big Data platform -- saving millions of dollars a year while providing a completely new level of insights to the business. This is a clear example of how mainstream organizations are using Hadoop. The “save costs and uncover new insights” combo seems to be a killer one. I expect more of these to dominate the 2014 scene.
4. The Establishment Will Tremble
What happens when disruptive technologies enter the game? The establishment trembles. I’m talking about the impact that Hadoop will have on traditional data warehouse and data integration giants (and, why not, mainframes as well). Let me be clear, the data warehouse, extract, transform, load (ETL) and the mainframe are here to stay. But the new “standard” architecture has already started the change and the impact will be significant to say the least.
Looking for examples? Well, I could point you to the latest earnings announcement of a leading data warehousing company. You can guess the results were not very encouraging. However, the most interesting part was the recognition that ETL workloads -- one of the primary use cases for Hadoop -- typically account for 20 to 40 percent of their database capacity. As some analysts quickly pointed out: Is it Hadoop or the economy?
There you have it, my list of trends for 2014. Part of the fun is knowing that no matter how much you know, time can always prove you wrong. Nevertheless, 2014 is shaping up to be a much more mature one for Hadoop and big data. A year where we start taking matters at face value, as we assimilate and operationalize the reality, leaving the hype aside … at least until the next big thing.
Editor's Note: Read another perspective on big data in Big Data Gets Sticky