Is big data darling Hadoop about to pack up its bags and leave home? Rumors and commentary are spinning around the web and traditional media that Yahoo! Is considering spinning off the entire software unit. Although the software originated with Yahoo!, Cloudera (news, site) has emerged as a leader in the paid support and value-added products for the platform. If Yahoo creates a Hadoop startup, Cloudera and "Yahoo spinoff" would compete head-to-head.

The Value in Hadoop

We previously reported about the shakeup in the continually emerging Hadoop market. The profitability of Cloudera, founded in 2008 -- which employs Hadoop creator Doug Cutting, who previously joined Yahoo in 2006 -- has driven over a dozen companies to offer Hadoop products and services. Now it seems that Yahoo! wants a piece of its pie back.

Yahoo has been a core supporter in development of Hadoop data. The Apache Foundation manages Hadoop, which is a top-level project, but Yahoo contributes a significant portion of the new code for the data processing platform by dedicating a large number of engineering resources. Yahoo uses the technology internally and other large internet services, such as eBay, Facebook and Twitter, also employ Hadoop.

How big is Hadoop? Big. Our need to store all of those social media interactions, location details and various other pieces of data isn’t showing any signs of slowing. Organizations need a way to analyze and understand it, and Hadoop has proven it plays well in the space. The Wall Street Journal reported that analysts expect Hadoop could be another billion-dollar business for Yahoo, possibly rivaling the $6 billion in annual revenue from the sale of online ads.

What will the new company do if it spins off? Many are predicting that Yahoo will not create a new distribution of Hadoop because it discontinued its custom “The Yahoo Distribution of Hadoop” back in January. It is more likely that company will focus on the profitable value-added services such as consulting, professional services and integration.

Despite all of the interest in Hadoop, as a platform the technology is still pretty raw. Many organizations simply don’t have the technical expertise in-house to leverage the platform. It is likely that, over the coming years, larger companies will go on spending sprees and consolidate the market down to a reasonable number of players.

Big Data, Content Management

What does this have to do with content management? As content sizes and volumes grow, and content grows more diverse and unstructured, organizations are being driven beyond traditional storage solutions like relational databases and file systems. Many are leveraging big data tools like Hadoop to interact with or back CMS platforms.

Yahoo has not commented on the spinoff, so who knows what will really happen, but we will be watching.