Two years ago, no one seemed to know what to do with big data.  Now, vendors are offering solutions that are intuitive enough to provide real business value, enable even non-data scientists to perform real analysis and have ushered in a new era of novel applications like data-driven design.

Big data is maturing — evolving from a frothy, over-hyped concept to a source of actionable insight.  It has captured the imaginations of companies of all sizes --including Ford, GE and Universal Music Group -- given a new voice to customers worldwide and has helped to create business value at a blistering pace.

Those were my main takeaways from the GigaOm Structure Data conference in New York City this week.

Like Water for Data

The most important thing to remember about big data is that it's a potential source of deep insight — and many companies still haven't fully leveraged its benefits.

New technologies, however, are making it easier and easier to derive value from that ever flowing data.

I like what ClearStory Data has developed over the past 18 months or so, which seems to be a version of Excel for the Enterprise with sharing capabilities.  At the conference Wednesday, the company unveiled new data harmonization, in-memory data processing and user model advances for faster data analysis across greater internal and external sources. The company claims the advancements "allow organizations to access, converge and harmonize dozens of disparate data sources on the fly for broad insights without requiring IT experts and long data manipulation cycles."

CEO and founder Sharmila Mulligan said the company's mission is  "to dramatically simplify accessing, exploring and analyzing data."

The big challenge I see with the product is getting various departments within an enterprise to write connectors to the ClearStory Data so the information can be curated in a useful way. The challenge isn't actually  writing the connectors, but knowing what should be connected and the best way to add texture to it.

Data-Driven Design

More organizations are using data to improve design. It's proving to be the fastest way to successfully test and implement new designs. At the conference Wednesday, Geoff McGrath, managing director of McLaren Applied Technologies, said the same design techniques used in automotive design can be applied to many other industries and products, from bicycles and baby strollers.

He described it as the convergence of data management, analytics and simulation to create high-performance design value at a fast pace. 

Translating the Data

The funny thing about big data is that it often helps to know what you’re looking for before you go looking. Take Crimson Hexagon, which offers social media monitoring tools that provide marketers with metrics and trends from earned and owned media. According to the company, 

Learning Opportunities

The only way to navigate this vast sea of meaningless information is to locate the Crimson Hexagon, the one room that contains a log of every other book in the library — a guide to extracting meaning from all the unstructured information. Crimson Hexagon aims to provide the same value to your business: extracting meaningful insight as the number and volume of conversations increase every day."

Crimson Hexagon's Topic Wheels provide a view of frequent terms and prevalent themes in a social media conversation and help make sense out of disparate data. Here is how it interpreted McGrath's talk:


It’s accurate based on my own memory of the talk. But if you don't believe me, here’s the Livestream of it so you can check yourself. Crimson Hexagon explains:

The inner rings represent frequently used terms and the outside rings provide a more granular look at words and terms and how they relate to the topics of conversation. With just a click, filter the Topic Wheel by sentiment or category to drill down further into one theme or topic of the online conversation."

Let's look at another Topic Wheel. This one looks at terms and themes around a conference presentation by Dennis Crowley, co-founder and CEO of Foursquare.


The promise of these text analytics tools, much as the other tools featured at the conference this week, is to give texture to the data. Remember: data, by itself, means next to nothing.  With this in mind, what I heard this week ought to give hope that the promise of big data to improve our lives is being realized, and that it is now closer than ever.

Title image by Nadly Aizat (Shutterstock).