Feature

Yahoo Entering the Commercial Hadoop Big Data Space

2 minute read
Josette Rigsby avatar

yahoo_logo_2010.jpg
Reports are circulating that Yahoo (news, site) will finally beannouncing its commercial Hadoop spinoff company to compete with Cloudera(news, site). In a nod to the competition, Yahoo is supposed to call the newcompany HortonWorks, for the elephant in the Dr. Seuss classic, Horton Hears a Who.

Hadoop Returns to Its Origins

As the amount of databeing produced has grown from megabytes to zettabytes, big data processingframework Hadoop has increasingly grown in popularity and created a new market forHadoop services, products and distributions with one of the largest providersbeing Cloudera. Yahoo hopes to position HortonWorks as the preferred vendor forproduction Hadoop distributions and make the Apache distribution the preferreddistribution for the core software. We reported back in May that Yahoo stopped development on its own Hadoop distribution andinstead focusing its effort on development of the Apache distribution.

Yahoo created Hadooptechnology and has contributed 70% of the code base to the Apacheproject -- although the project’s originator, Doug Cutting, now works forCloudera. Yahoo’s entry into market is a natural move for the company and thecompany’s entry into the market should have a large impact on market evolution.The spinoff, HortonWorks, will consist of a small number of Yahoo engineerswho will concentrate on creating a production-ready release based on ApacheHadoop. The HortonWorks Hadoop distribution will include new features, such asmanagement tools, on top of the core aimed at making Hadoop easier to use.  It is believed that the value-added tools developedby HortonWorks will be open source and Yahoo will work closely with Apache asthe product develops.

Learning Opportunities

The Hadoop Market

The Hadoop market has grown substantially. There seems to bea constant stream of new entrants offering new services such as DataStax (news,site) and EMC. However, Yahoo’s entry into the commercial Hadoop space willdefinitely increase the competition. The Hadoop market is still relatively small in spite of growing levels ofadoption. Although the market is still somewhat small, many project it willgrow into a multi-billion dollar opportunity with growing levels of competitionfrom new big data alternatives.

The announcement of the Yahoo spinoff should come today ortomorrow if reports are correct. How do you think the entry of Yahoo into theHadoop space will affect the market? We would love to hear your thoughts.