There’s almost as much buzz around Big Data as there is around Pinterest. Chances are that most anyone you run into at a premium coffee shop in an upper middle class neighborhood will have heard of, and be curious about, both.
And while Pinterest is the world’s fastest growing, stand-alone internet site of all time, it’s that way because it’s full of attractive images and it’s geared toward consumers. You use Facebook to log-in. check a few boxes and voila, appealing images curated especially for you appear on the screen in front of you.
It's Just Data
Big Data is quite a different story. Sure, it’s a term that our friends and our neighbors feel familiar with; after all, there have been numerous articles about it in Business Week and Forbes. The same can be said of Hadoop (NOTE: it’s a software framework that supports data-intensive (aka Big Data) distributed applications).
But the truth is that most people don’t have much of an appreciation for what Big Data actually consists of -- petabytes, exabytes or zettabytes of data gleaned from web-clicks, smartphones, telemetric devices, iPads, personal computers, ultrasound scanners and anywhere that digitized data can be gleaned from. If thinking about it all gives you a headache and you still don’t understand what Big Data is, don’t worry about it, “It’s just data,” Hilary Mason, the chief scientist at bit.ly once said.
Tomorrow's Rock Stars
Simple as that sounds, it doesn’t mean that everyone who has gotten a B-minus or higher on a stats or calculus test should be urged to go into Big Data by their well-meaning parents or guidance counselors. You can’t blame them for wanting their offspring to become the data scientists who discover the gems hidden in a data stream. It’s this generation’s version of “Go West, young man,” and it’s predicted that the geeks will find the Holy Grail and/or strike gold. (Some of them already have.)
Working with Big Data is supposed to be sexy too, at least if you buy into EMC’s prognostication that Data Scientists will be tomorrow’s rock stars. Most of the data scientists that I’ve interviewed either shrink or laugh when I ask if they feel like rock stars. Sure, there’s a cool factor, but they went into the field long before terms like Big Data or Data Scientist even existed.
In fact, today’s data scientists usually became data scientists while they studied science or math or even the arts with a determination of understanding their world and its less obvious patterns of connectivity. They typically love asking questions and creating equations to help them make discoveries. They design visuals to help them explain what they’ve learned. They’re comfortable with uncertainty and they call data beautiful with the same sense of appreciation that Martha Stewart has for place settings, that Mario Batali has for pasta and that the Grateful Dead had for jamming.
Though some data scientists sound like they love their jobs so much that they’d do them for free, the truth is that most of them get paid rather handsomely. And that’s not likely to change. Data scientists generally have Master’s degrees or PhD’s, so it takes at least six years to make them and the demand for them is expected to grow exponentially.
Where They Work, How They Work
Today most data scientists work for two kinds of employers, internet firms like Facebook, Linkedin, Amazon, Google and the like or for Wall Street firms like Goldman Sachs, Bank of America, Morgan Stanley, et al…(though at some financial firms, some of them are called quants).
In the not-so distant future, non-web based and non-investment based companies are expected to use Big Data to reinvent everything they do from CRM to internal processes to product design. (In fact we’ve already talked to people who are doing this; they will be featured in my follow up.)
Many of today’s IT professionals are going to have to rewire their brains and learn to retrain users if they plan to work as data scientists. “Big Data turns the process of decision making inside-out,” says Professor Alex “Sandy” Pentland, Director of MIT’s Human Dynamics Laboratory and the MIT Media Lab Entrepreneurship Program. He explains that normally a problem is approached with some understanding or guess about what to do, and “then you construct a test of your hypothesis in order to validate it, and statistics tell you if it’s true or false.”
With Big Data you don’t start with a hypothesis, according to Pentland, instead you mine patterns from the data.
Sound interesting? In the next installation you’ll read about how a variety of businesses and organizations are using Big Data to create new products, new pricing models, improved customer experiences and tools for public good.
Editor's Note: You may also be interested in reading: