Who Says 'Big Data Needs to Shrink to Grow'?While most people were busy nursing their New Year’s Eve hangovers or getting busy with their resolutions on January 1, the New York Times ran a rather interesting headline: "Big Data Shrinks to Grow." We looked at it and said, really? That’s not our experience, but continued to read, anyway.

The article states that interest in big data is waning; the author bases his claim on the fact that Google searches for the term “big data” are no longer rising. We think that may be a pretty lousy basis for an argument; after all, it could be that the reason that searches for “big data” are down is because many people already know what it is. (As I stated in my year end big data wrap-up “If 2012 was the year your grandmother instigated big data conversations at the dinner table (yes, the 'buzz' around it actually was that big) then 2013 will go down in history as the year the enterprise began to make serious plans around it.")

We also hope that enterprises don’t craft their big data strategies via Google search. But enough about that.

The article also points out that Kaggle (a site that hosts data scientist competitions) has changed its business model from one that spans the marketplace to one that specializes in specific industries, starting with Oil and Gas.

Interesting? Yes, but a trend? We're not so sure. Hopefully data scientists, statisticians or even high school students with a little common sense will point out that one or two companies changing their business strategies does not a market trend make.

The article does bring up more interesting questions like: do data scientists and other professionals who work with big data need to have deep industry insight to deliver discoveries that warrant the costs of wrestling with big data in the enterprise?

It quotes Kaggle founder Anthony Goldbloom saying:

We liked to say ‘It’s all about the data,’ but the reality is that you have to understand enough about the domain in order to make a business. What a pharmaceutical company thinks a prediction about a chemical’s toxicity is worth is very different from what Clorox thinks shelf space is worth. There is a lot to learn in each area.”

Goldbloom makes a good point, but does this mean that data scientists who don’t have industry specific training can’t yield the returns that big data hype suggests?

If so, then the big data business may be in serious trouble because data scientists are rare enough, add another skill to their heavy list of “must have requirements” and we’ll not only have to wait for them to finish their post graduate training but to also get jobs and work for five years before they can add value.

The Experts Weigh In

Enough about what we think. We asked four firms that provide products and services to enterprises to comment on questions like: Does big data need to shrink to grow? and Do data scientists need to have industry specific experience to be worthy of their hefty price tags?

Here’s what they said:

On the question as to whether data scientists need to have domain knowledge to make cost justifiable contributions Sandy Steier, CEO and co-founder of 1010data said:

To add value to any industry, a person would presumably need a certain amount of both analytical expertise and domain knowledge. An interesting question is, is it better to start with domain knowledge and learn big data analysis from there, or is it better to start with analytical experience and then apply it to a new domain?

I believe the latter is the easier path to success. I'm sure some domain experts would want to emphasize domain knowledge, but in my 35 years of analytical experience and 14 years of growing 1010data, that's what I have seen pretty consistently. Certainly the normal path is to be schooled in analysis or programming, get a job in a specific industry, and then perhaps even change industries."

Byron Banks, Vice President, database & technology at SAP offered a different point of view: