What You Need to Know About Big Data Hype

By Cheryl McKinnon (@cherylmckinnon) Feb 23, 2012

The backlash against the buzzword of the year — big data — to some extent has begun. As Forrester's James Kobielus so eloquently put it, “Some shiny new thing gets built up until it’s too big for its britches and then we delight in shooting it down.” Practitioners who live and breathe content and information management technology on a daily basis can become tired of the hype behind new labels and acronyms as they get more and more attention from technology marketers and pundits.

But mild annoyance with an overused label does not negate reality. “Big Data” is something very different for most information professionals. Starting the learning curve now — rather than when a crisis or new initiative pops up out of nowhere — is what practitioners in the IT, compliance and business process automation arenas need to do.

Why is “Big Data” Different?

The “Internet of Things” opens the door to a whole new world of electronically stored information (ESI)

One aspect of big data is the tremendous volume of new digital activity being collected, logged and potentially analyzed from a new generation of digital devices. From GPS-enabled phones and vehicles, to sensors and monitoring tools with their own IP addresses, data is being generated, collected and stored everywhere we look.

Utility systems, transportation systems and retail systems are all collecting information about how we use resources, shop and travel. Information professionals who focus on compliance, privacy and retention policies need to recognize this new source of ESI and include it in their corporate data mapping as a potential source of e-Discovery requests.

Big Data infrastructure applications were created to solve problems that haven't existed before

New database architectures and software frameworks have emerged, designed specifically for data heavy applications that are distributed across countless nodes and servers, storing vast amounts of data. The rise of social business, mobile transactions and cloud services has accelerated the need for a new approach to data scaling and storage.

Most of these new frameworks and databases have their roots in the open source world, where developers routinely create new approaches to problems that haven't hit mainstream. Companies such as Yahoo, IBM, Apple, Amazon, Twitter, eBay — the companies that represent many of the biggest providers of online communication and transactions we use both as consumers and professionals use and contribute to these innovative, open development initiatives. As software philosopher Eric S. Raymond once said, good software starts by scratching a developer's itch. New problems demand new solutions. New scale of digital content, information and communication demands new architectures.

Other people's data can help your business

There is a school of thought that states that in 2012 only a handful of the very largest global organizations or popular social network platforms are really facing the big data problem. But then we can recall the famous prediction in the late 1950s, that “there is a world market for about five computers.”

Innovation is driven by creatively applying new tools to new problems, thereby discovering new opportunities. Many companies will not, in fact, have in-house big data issues in the short term. But access to data from vast social platforms for research or marketing purposes, often via APIs, means big data can be used for better business decisions.

Optimizing lead generation campaigns, analyzing content for sentiment or trends, and getting real-time insight into an industry rather relying on old historical data can be tremendously valuable. C-level executives need to start thinking about how access to such data can be used to improve analytics, reduce costs, find new revenue opportunities and extract value from masses of information.