Reducing the Risk of the Data LandfillThe big data revolution has led to an exponential increase in the volume, velocity and variety of information available to businesses for use in decision making. It has also greatly complicated the work lives of business analysts who are expected to turn mountains of data into actionable insights for their marketing departments.

These professionals now have to deal with a data landfill -- mountains of data that enterprises are collecting and trying to use to make business decisions. This fast changing world requires a new strategy that goes beyond the current data preparation methods used by IT.

The Current Challenge of Data Preparation

Data analysts have always faced difficulties in preparing data for analysis. There is almost always a diversity of sources, both internal (such as sales, manufacturing and finance) and external (such as third-party providers, public sources and the Internet). Data also comes from a variety of locations and in various formats, such as Excel, JSON and XML.

To date, analysts have spent the vast majority of their time preparing data and much less time doing actual analysis, a problem that has only been exacerbated by the challenges of getting these rapidly growing data sets into the right form. A recent InformationWeek study on big data reported that 59 percent of respondents said data quality problems are the biggest barrier to successful analytics. More than ever, data preparation is a significant impediment to informed and timely decision-making for marketing departments looking to take advantage of their big data.

A recent article in the Harvard Business Review reported research on best practices for how businesses should use data. The article reported that companies with a culture of “evidence based decision-making” consistently see improvements in their business performance. One of the hallmarks of such companies, the research found, was that they “ensure that all decision makers have performance data at their fingertips every day.”

The implications for data preparation are clear: It cannot be a month long process handled exclusively by IT professionals. Basing a campaign off of data that old might have been acceptable 10 years ago, but today’s business analysts need to conduct analytics quickly if they want to keep up.

More than ever, marketing departments need to be able to perform their analytics ad hoc.

Envisioning a New Method

As more data is collected from more sources, IT departments have begun to create data landfills. These landfills can be rich repositories -- but only if users can reach the “right” data easily and quickly. Having the right data is one of the biggest challenges for envisioning a new method of data preparation.