EMC Greenplum Open Sources Big Data Apps, Acquires Pivotal Labs

4 minute read
Virginia Backaitis avatar

“What Java did for the Internet, they (EMC Greenplum) will do for the web.” That’s a pretty profound proclamation, especially when you consider who’s making it; none other than Scott McNealy, the co-founder of Sun Microsystems, where Java was created in the early 1990’s.

McNealy was responding to two announcements made by EMC Greenplum yesterday. First that it would be Open Sourcing its Greenplum Chorus product later this year, and second that it had acquired Pivotal Labs, a software co-development consultancy that has helped companies like Twitter, SalesForce, Groupon and BestBuy leverage their data assets to the hilt.

Open Source Big Data Apps

For those who aren’t familiar with Greenplum Chorus, “It’s like Facebook for Data Scientists” say Pat Gelsinger, President and Chief Operating Officer, EMC Information Infrastructure Products. The Chorus social platform eases collaboration between data science team members so that they can easily and expediently share datasets, discoveries, insights and results as they work.

The outcome? More, high quality, Big Data apps (that just happen to be built on top of the Chorus platform). Why Open Source it? Because more developers will have access to it, will improve it and enthusiastically build apps on top of a platform that isn’t proprietary. And, of course, the more apps that are built on top of Chorus, the more powerful and valuable Chorus becomes. Think of Android on top of Google or Zynga on top of Facebook. It’s the Apps that matter.

Building products and applications that leverage Big Data is not easy; Greenplum found this out while they were trying to build Chorus."We had this set of ideas around Chorus (and) it ended up being a challenging thing for us to write all of these things,” says Scott Yara, Greenplum’s Senior Vice President, Products and one ofits co-founders.

So rather than miss deadlines and struggle, Greenplum called Pivotal Labs for help. They were pretty impressed with their experience and their results. “We liked the razor so much, we bought the company,” jokes Greenplum Senior Director, Product Marketing.

Two Heads are Better Than One

Pivotal Labs certainly works differently than traditional consultancies. Rather than do some analysis, set up a project plan and deliver a solution, the company brings its clients into their facilities (presently they have offices in San Francisco and New York City, but as part of EMC, they’ll be going worldwide) and work with them to leverage datasets and build applications side by side. (It’s why Pivotal is known as a co-consultancy.)

Learning Opportunities

The company’s engineers use an agile methodology which is commonly based on incremental development where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams.

At Pivotal, this means that say, for example, ten engineers are assigned to a project; five development teams of two are then created. Each pair of engineers sits closely together and shares a table and a screen view. Every line of code that is written is blessed by two engineers.

The idea is that there is space for more than one idea, that the engineers learn from one another as they work, and that their end product is superior to that whichone person would have built alone (a.k.a. “two heads are better than one.”) Add to that that the individuals working on a project rotate so that no two engineers work together for a long period of time.

This methodology, Pivotal’s well-pedigreed engineers, and their Big Data app development expertise have the potential of producing big wins for EMC Greenplum and its customers.

It’s not so much that Pivotal’s service revenues will add to EMC’s bottom line in a big way, but that a few years from now there will be a good number of brilliant, skilled developers and Data Scientists who have one big thing in common -- they will all have been indoctrinated via Greenplum’s database, Greenplum’s Hadoop distribution, Greenplum Chorus and Pivotal’s agile methodology.