Pivotal Revs Its Big Data Play But Theres a Better Story

And Pivotal’s big data platform and Pivotal GemFire, in particular, is powering some of them.

Pivotal GemFire, for anyone who needs a refresher, is a distributed in-memory data management solution for enterprises creating high-scale custom applications.

Since real world examples are sometimes worth more than a million words, we’ll put the geeky stuff aside for now, but we’ll get back to it.

Travel Issues, On a Massive Scale

Picture this: the largest annual human migration on the planet. It’s called Chunyun and it happens in China over a 40 day period around the time of the Chinese New Year. During it 3.62 billion trips are made, many of them by train. Not going home -- which is often a long, long way away during this time period -- isn’t much of an option and there are no excuses.

So what happens when everyone tries to purchase train tickets at once, when servers go down leaving some workers without train tickets and others double-booked? Train stations aren’t happy places. There are fights between people who have been sold the same seat and hearts break when migrant workers aren’t able to get back to their families.

It’s a Big Data Problem

The picture is not pretty and the solution didn’t seem simple. After all, the China Railway system needed to find a way to handle 4.5 million ticket purchases and 20 million users per day. We’re not talking about people standing in lines or sitting at home on their PCs -- these are migrant workers who use smartphones for the task.

And while there’s that to consider, there’s something else too: many of the railway’s customers work shifts, so any solution considered had to be able to scale spikes of 15,000 tickets per minute and 40,000 site visits per second without any downtime.

This is a big, big data problem.

There’s a Big Data Solution

We always hear that today’s problems can’t be solved with yesterday’s technologies and this is a perfect case in point. Because while Chunyun is not new, smartphones are. So when China Railway tried to provide ticketing services generated in a third platform world using second platform solutions, it didn’t work well. Servers were failing, there was no single version of the truth, coherence was lacking and there was havoc.

And while it’s simple to say that China Railway implemented Pivotal Big Data Suite’s GemFire and the problems were gone, that’s pretty much what happened.

“The system is operating with solid performance and uptime. Now, we have a reliable, economically sound production system that supports record volumes and has room to grow,” says Dr. Jiansheng Zhu, Vice Director of China Academy of Railway Sciences

Pivotal GemFire 8, Built for a Third Platform World

For anyone who’s not already familiar with Pivotal GemFire, it’s the transaction (OLTP) component of Pivotal’s Big Data Suite. It’s made up of tools that help developers build reliable, high performance, big data apps at scale while always being “on,” such as the ticketing solution at China Railway.

Today Pivotal announces a new release, GemFire 8 that offers new tools and capabilities to enterprise architects and developers and strategic advantages to the business.

More specifically GemFire 8 provides:

  • In-memory compression: Speed-optimized in-memory compression, allowing individual nodes to manage up to 50 percent more data per node than before
  • Resilient automation and rolling upgrades: Automatic node reconnection and data restoration, and a new ability to serially update software on nodes in a cluster that remains live -- eliminating a need for planned downtime for upgrades
  • A new restful API: Developers can enhance the performance and resilience of a wider range of high-scale applications such as those developed in Ruby, Scala or Node.js computer languages

Defining the Third Platform 

When Pivotal first launched in late 2012, it set out to define how big applications and apps would be built on the third platform in an era characterized by big data, mobile, social and cloud.

The company pooled assets from EMC and VMWare to create its Big Data Suite in which all information is stored, where data scientists and business analysts can discover insights that might lead to significant strategic advantages, and can then be operationalized within applications.

Has Pivotal Software Now Changed the World?

When Greg Chase, Pivotal Software’s director of product marketing, announced that he had joined the company last April, he said he did so to help build a company that “you know can change the world.”

His colleague Sudhir Menon, R&D director at Pivotal’s GemFire, may very well suspect that that’s already happening. Between the railway ticketing solutions that GemFire powers in China and India “we move one-third of the world’s population,” he says.

It seems like a pretty big deal.

And when it comes to helping Pivotal boss Paul Maritz write the story of how application development, deployment and operations is going to be reinvented, the GemFire team and the team behind Pivotal’s Big Data Platform may be well on its way.

“We have a unique portfolio and we’re outperforming expectations,” says Michael Cucchi, senior director of product marketing at Pivotal. He adds that the company’s solutions do other remarkable things as well, like helping clients win business and make more money simply because of the speed of its platform.

Title image by Junyu Wang (Flickr) via a CC BY-NC-SA 2.0 license