I really do like it when mainstream media pick up on trends that folks like us have been talking about for some time. To me, it validates its significance, even if some of the topics need to be simplified to appeal to the masses. However, sometimes it annoys me when those in the media get it wrong. Case in point: on Monday, Dennis Overbye for the New York Times wrote a seemingly harmless human interest story about big data.

How Big is Big?

In the article, Mystery of Big Data’s Parallel Universe Brings Fear, and a Thrill, Overbye asserts that

"Big Data probably knows more about us than we ourselves do, but is there stuff that Big Data itself doesn’t know it knows? Big Data is watching us, but who or what is watching Big Data?

It’s easy to get confused when the term big data is thrown around as much as it is. But when we talk about big data, what do we really mean?

According to the folks at IBM, big data spans three dimensions: volume, velocity and variety, and can be used as an opportunity to “find insights in new and emerging types of data and content, to make your business more agile and to answer questions that were previously considered beyond your reach.”

To this end, big data is supposed to know more about us than we do. The human brain is very complex, but it can’t begin to process the amount of information we’re creating every minute. Overbye seems to be impressed when he is told by a science historian that “the world’s bank of digital information, growing at a rate of roughly five trillion bits a second." I’m not dismissing that five trillion doesn’t sound like a lot. But data is growing at a far faster speed and greater rate than that. Consider the following:

  • 2 terabytes of Tweets are created each day (IBM)
  • 7 exabytes of data were stored by companies in 2010 (IDC)
  • In 2011 alone, the amount of digital information created and replicated surpassed 1.8 trillion gigabytes (IDC)

How Big is Too Big?

The rest of Overbye’s article does raise some interesting points about the existentialism of big data. Can it get to the point when the data we’ve created is going to overtake us? While it’s always a possibility, I suppose, the problem with big data isn’t how it can be used against us, but how we view big data in the first place.

In a white paper by Forsyth Communications aptly titled “For Big Data Analytics There’s No Such Thing as Too Big: The Compelling Economics and Technology of Big Data Computing,” the authors write:

"The astronomical numbers relating to the size of the digital universe make a compelling case for organizations to accept that what they’ve been doing with their traditional relational databases and enterprise data warehouses (EDW) in order to remain competitive up to now just isn’t going to cut it much longer. That is, if they’re going to take advantage of the huge new benefits of big data, avoid the headaches, and turn it into real business value. This means shifting the perspective from data as structured and disciplined to extracting value from the vast, messy array of unstructured digital data engulfing us; a rethinking of the data deluge as a problem to the data deluge as an opportunity.

There’s no denying that we’ve got big data. And while not all big data is good data, instead of appeasing it with a smirk and a shrug, we should get it know it better. There are already so many of us doing this already. For marketing analysts and data scientists alike, being able to understand what our data is saying so we can better engage with others, plan smarter for future events, and manage it more effectively is what we strive to accomplish. Big data will only be our undoing if we fail to see the opportunity it provides us and our future.