big data failBig data is on the minds of just about everyone, with IT departments large and small grappling with exponentially growing volumes of both structured and unstructured data. But despite big data’s place as a mainstream IT phenomenon, the bulk of big data projects still fail, as organizations struggle to find ways to capture, manage, make sense of and ultimately, derive value from their data and information.

Taming big data and being able to get the business insight you need is a daunting task all by itself; but if everyone isn't on the same page when it comes to defining the scope of the project and all the right pieces aren't in place, the project is destined to fail.

Causes for Failure 

These are what I see as the main causes of failure for big data projects:

  • Lack of alignment. Business and IT groups are not aligned on the business problem they need to solve but instead are tackling it from a technology perspective. Lack of true commitment from business stakeholders also makes alignment harder to achieve.
  • Lack of access. Access to data often is restrictive, and team members don’t have access to the data sets they need to find answers that will make the project successful.
  • Lack of knowledge. Many of the technologies, approaches and disciplines around big data are new, so people lack the knowledge about how to actually work with the data and accomplish a business result.

Lack of Alignment

Of all of these pitfalls, the first -- lack of alignment on the business questions you’re trying to answer -- is the most important. The whole idea is that you are exploring and searching for what you don’t know; so, to achieve success, it’s critical to define the project in terms of exactly what the business is trying to accomplish and what questions need to be answered.

Although it’s the most important factor in the success of a big data project, alignment is challenging to achieve. Not only does big data mean different things to different people, but a host of external factors can influence changes in business requirements and priorities faster than IT can keep up. If IT and the business aren't aligned on the scope of the project, it can shoot off in too many directions, involving too many people and shifting the focus from answering specific business questions to managing the technology needed to achieve everybody’s piece of the pie.

Another challenging impact on business/IT alignment comes from unwillingness to change. Too often, if a big data project suggests an action or change that feels too foreign to business stakeholders, they may be reticent to accept it, dismissing it as a faulty process, analysis or data-set. In response, analyst groups may steer results for a future project in a direction they think the business will agree to and act upon, resulting in recommended actions that may create a sub-optimal business outcome.

Lack of Access

The second reason big data projects fail -- lack of access to data -- goes back to a fundamental IT premise: silos. There are data silos for sales, marketing, HR and others, each restricted and guarded to meet compliance. There are good reasons why data silos exist but if the data you need is not available to you, you’re limited before you even start to solve the problem.