Big data is on the minds of just about everyone, with IT departments large and small grappling with exponentially growing volumes of both structured and unstructured data. But despite big data’s place as a mainstream IT phenomenon, the bulk of big data projects still fail, as organizations struggle to find ways to capture, manage, make sense of and ultimately, derive value from their data and information.
Taming big data and being able to get the business insight you need is a daunting task all by itself; but if everyone isn't on the same page when it comes to defining the scope of the project and all the right pieces aren't in place, the project is destined to fail.
Causes for Failure
These are what I see as the main causes of failure for big data projects:
- Lack of alignment. Business and IT groups are not aligned on the business problem they need to solve but instead are tackling it from a technology perspective. Lack of true commitment from business stakeholders also makes alignment harder to achieve.
- Lack of access. Access to data often is restrictive, and team members don’t have access to the data sets they need to find answers that will make the project successful.
- Lack of knowledge. Many of the technologies, approaches and disciplines around big data are new, so people lack the knowledge about how to actually work with the data and accomplish a business result.
Lack of Alignment
Of all of these pitfalls, the first — lack of alignment on the business questions you’re trying to answer — is the most important. The whole idea is that you are exploring and searching for what you don’t know; so, to achieve success, it’s critical to define the project in terms of exactly what the business is trying to accomplish and what questions need to be answered.
Although it’s the most important factor in the success of a big data project, alignment is challenging to achieve. Not only does big data mean different things to different people, but a host of external factors can influence changes in business requirements and priorities faster than IT can keep up. If IT and the business aren't aligned on the scope of the project, it can shoot off in too many directions, involving too many people and shifting the focus from answering specific business questions to managing the technology needed to achieve everybody’s piece of the pie.
Another challenging impact on business/IT alignment comes from unwillingness to change. Too often, if a big data project suggests an action or change that feels too foreign to business stakeholders, they may be reticent to accept it, dismissing it as a faulty process, analysis or data-set. In response, analyst groups may steer results for a future project in a direction they think the business will agree to and act upon, resulting in recommended actions that may create a sub-optimal business outcome.
Lack of Access
The second reason big data projects fail — lack of access to data — goes back to a fundamental IT premise: silos. There are data silos for sales, marketing, HR and others, each restricted and guarded to meet compliance. There are good reasons why data silos exist but if the data you need is not available to you, you’re limited before you even start to solve the problem.
To get past that, big data projects have to start with executive-level ownership. Without all the relevant data from the business, it’s impossible to see the relationships and patterns that will answer the business questions, so buy-in has to come from the top. Somebody at the top has to say, “This team is looking to solve a specific business issue and is important enough to have access to any data they need.” Without access to the right data, the project will be at a standstill. Period.
Lack of Knowledge
The third pitfall — lack of knowledge — is about having the right people who have the right skills to execute the project. You need people who understand machine learning and natural language processing to name just two. Because this technology is so new to the "mainstream," IT teams frequently lack those who understand how to use it for analysis purposes.
While hiring a data scientist is one possibility for addressing this knowledge shortfall, it’s not feasible for many organizations. This new role combines the mindsets of scientist and investigative researcher with the skills of a programmer, but it’s an expensive position to fill and the required skill set is neither commonly found, nor easy to create.
How to Succeed in Your Big Data Project
Consider a practical approach. To start with, don’t call it a “Big Data Project.” Call it something like, “a project to learn more about our customers and why they shop in a particular store.” The project answers important business questions, and big data is the source for the answers. Here are some best practices to help your project realize a successful outcome:
- Why Agile As We Know It Will Disappear
- SWAM: When LinkedIn Locks Down Social Networking
- The Metamorphosis of the Social Enterprise
- Intranet Search: Where Documents Go to Die or KM Enabler?
- Just How Badly Does Microsoft Want Your OneDrive Biz?
- Pivotal Revs Its Big Data Play, But There's a Better Story
- ROI Is the Wrong Tool to Justify Social Investments