If you really listen when people talk, you can learn some amazing things. And in an age where chatter about the Cloud, Big Data and the shortage of Data Scientists has gone viral, there’s a quieter, compelling conversation going on. 

The next wave of computing will be about Big Data App development.

And it can’t arrive until Big Data App development is democratized. In other words, when you won’t need to be an elite geek who has a Ph.D. from an Ivy League school and easy access to the resources of a Facebook, Google, LinkedIn or some other hot property to write the programs that will shape how we work and do business in the future.

Though this article is about why EMC and VMWare are pooling their assets to create Pivotal Initiative,I want to credit the founders of stealth startup Continuuity for first enlightening me on how truly difficult Big Data App development can be at this point in time.

“It’s really, really hard,” they said, despite their fancy pedigrees and their records of accomplishment at places like Facebook and Yahoo in creating Big Data Apps. They knew that before more people could do it, it would have to become easier.

So they’re creating a company to help make that happen.

Pivotal Initiative: Writing the Book on Big Data App Development

Though no one has explicitly told me this, Pivotal Initiative (PI) is up to the same thing. The VMware EMC spinoff is being configured to solve the same problem and, perhaps, a few more. Consider what Paul Maritz, who will lead PI said before a VMWorld audience earlier this year (not long before he stepped down from his position as CEO).

Referring to VMware, Maritz said,

The story of the Software Defined Data Center (SDDC) has been written. We are not going to be the only vendor of a SDDC, but all of us who are going to provide one, understand how this is going to end.”

He then added (listen up here),

The story of how application development, deployment and operations is going to be reinvented is just as compelling, just as important, but has not been written. The participants in the creation of this story do not know how it will end.”

And though he stopped short of saying “that’s the story I’d like to play a lead role in,” he didn’t. Maybe, at that time, he wasn’t sure if this would happen at VMware, in another part of the EMC world that Joe Tucci controls, or somewhere else altogether.

Now he knows, and we do too.

Pivotal Initiative is going to be all about the developers, baby!

Yesterday EMC and VMWare jointly announced Pivotal Initiative via (once again) a blog post. We had reported that there’d be some sort of spinoff that involved EMC Greenplum, Pivotal Labs, and VMWare on Monday.

EMC Broadens its Horizons

With hindsight being 20/20, it’s obvious that the leaders of EMC Greenplum and Maritz have been gnawing on Tucci’s ear for some time to make this happen.

Learning Opportunities

Consider that EMC launched Greenplum Open Sourced Chorus, a Facebook like app for Data Scientists and Big Data App developers, last March. At the same time, they announced that they had purchased Pivotal Labs, a Big Data app consultancy which works hand-in-hand with customers to build solutions (and teaches them to do so independently in the process). Interestingly, Greenplum met Pivotal when they found out that they couldn’t build Chorus on their own because it was too hard.

Next, Greenplum introduced an analytics workbench.

Then they announced a partneship with Kaggle to bring to market an on-demand data science workforce.

Could Greenplum have told the world that “We need more people to build Big Data Apps so we’re going to make Big Data development easier” any more clearly?

But throwing tools at developers may not be enough, that’s why Pivotal Initiative is grabbing-up some properties that Maritz purchased while at VMware; namely, the Cloud Foundry Platform as a Service (PaaS), GemStone data caching technology, SpringSource Java frameworks, among others.

The EMC Greenplum/VMware mashup provides developers with everything they need to focus on building Big Data apps. And the Open Source nature makes it somewhat of a free-for-all.

It’s a nice vision and a huge leap for EMC, which seems to be broadening its line of sight from being a storage company with proprietary products to include Open Source and Software.

As Matt Asay of 10gen, the MongoDB company, told me in a recent interview, “Developers are the kingmakers.”

Time will tell if Maritz’s Pivotal Initiative will wear the crown.