The earth may rattle and jaws may drop in Silicon Valley this afternoon when EMC Greenplum introduces Pivotal HD, the world’s most powerful Hadoop distribution in history. (The latter according to EMC Greenplum).

And, sure, it seems like a bold claim to make, but it is well-founded. Pivotal HD’s Hadoop distribution natively integrates EMC Greenplum’s super-fast, industry leading, massively parallel processing (MPP) database with Apache Hadoop, which is widely acknowledged as the most cost-effective and flexible open source Big Data platform ever developed.

The integration leverages EMC-developed and patented “Dynamic Pipelining” technology which brings up to 100 times performance improvement in certain classes of query processing, according to EMC.

What this means for Enterprises and data workers is instant access to the Big Data-rich, real time information needed to make critical and competitive decisions on the fly. Up until now, regardless of various vendor claims, many enterprise data workers have found it frustrating to write queries that provide meaningful insights in a timely manner.

Some say that this is the reason why many Hadoop-related projects in the Enterprise space haven’t gone much beyond the pilot stage. Would-be adopters commonly claim that Hadoop platforms are hard to use, that getting anything done requires developers, coders and other technical experts who are expensive and in short supply, and that data workers and other users haven’t had the interfaces or tools needed to reap enough benefit for the effort required.

Pivotal HD was created to change all of that.

What Pivotal HD Offers

It promises to remove these and other barriers to widespread adoption through not only scalability and speed, but also by eliminating steps like moving data between systems or using connectors that require users to store data twice. Add to that that by providing the world’s first true SQL processing for Hadoop, many of today’s SQL trained analysts and users of BI tools will be able to analyze data sets stored in Hadoop from Day One. No additional training required.

As opposed to most other solutions on the market, Pivotal HD eliminates the data movement challenge, requires less data storage, the data is fresher and data management is simpler, says Josh Klahr, Vice President, Product Management, EMC Greenplum. Though there may be at least one other Hadoop vendor who promises to do the same, they’ve been working at building MPP for under two years; EMC Greenplum has been at it for over a decade.

Not only that, but EMC seems to be betting that all of tomorrow’s data will be stored in a single repository. “It’s a big strategic bet,” says Klahr, but he’s confident that it’s the right one.

EMC’s Big Data news doesn’t stop there.

The Next Management Revolution

Paul Maritz who appears to be Pivotal Initiative’s (EMC’s newly spawned company which will likely be in stealth mode for a few more weeks) boss-to-be recently told Wired Magazine that his new company will aim to be “Google in a Box” for the Enterprise. Pivotal HD will, no doubt, play a huge role in that, though Greenplum refused to confirm that Pivotal HD will serve as the Pivotal Initiative (PI) data repository. (They did promise that more on PI would be revealed in March.)

In the article, Maritz also confirmed that Robert Mee, president of EMC owned application co-development consultancy Pivotal Labs, would be taking charge of Cloud Foundry, a company that EMC brought to PI from VMWare. Cloud Foundry is a platform-as-a service play that allows companies to build software applications that scale amongst a massive number of users.

It does not seem like too far a leap to make to assume that Mee will be charged with shoring up Cloud Foundry, thereby enabling Pivotal Initiative’s clients to build applications (with or without the help of Pivotal Labs) that will likely leverage data stored in Pivotal HD.

It’s worth noting too, that Mark Lucovsky who played a significant role in building Google’s search API’s will be moving from VMware to Pivotal Initiative where he will… just take a guess.

Finally, late last year when many journalists and analysts suggested that Pivotal Initiative would be a dumping place where Maritz would try to mash-up Greenplum and some of VMware’s misfit acquisitions, we said that nothing could be further from the truth.

Though we don’t know everything about Pivotal Initiative yet, one thing is already certain: Pivotal HD is breaking new ground and introducing a better way for Enterprises to work with Big Data; to ask and get answers to questions that were impossible, or impractical to ponder in the past; and to get better business results by becoming data-driven organizations.

This may be a pivotal point in the creation of, what Harvard Business Review authors Andrew McAfee and Erik Brynjolfsson call, the next Management Revolution.