To get past that, big data projects have to start with executive-level ownership. Without all the relevant data from the business, it’s impossible to see the relationships and patterns that will answer the business questions, so buy-in has to come from the top. Somebody at the top has to say, “This team is looking to solve a specific business issue and is important enough to have access to any data they need.” Without access to the right data, the project will be at a standstill. Period.

Lack of Knowledge

The third pitfall -- lack of knowledge -- is about having the right people who have the right skills to execute the project. You need people who understand machine learning and natural language processing to name just two. Because this technology is so new to the "mainstream," IT teams frequently lack those who understand how to use it for analysis purposes.

While hiring a data scientist is one possibility for addressing this knowledge shortfall, it’s not feasible for many organizations. This new role combines the mindsets of scientist and investigative researcher with the skills of a programmer, but it’s an expensive position to fill and the required skill set is neither commonly found, nor easy to create.

How to Succeed in Your Big Data Project

Consider a practical approach. To start with, don’t call it a “Big Data Project.” Call it something like, “a project to learn more about our customers and why they shop in a particular store.” The project answers important business questions, and big data is the source for the answers. Here are some best practices to help your project realize a successful outcome:

Start with a list of questions and business problems you want to answer

Don’t tackle something huge. Start with a small project, make it about a specific issue you need to address and stick with it. Make a list of the questions you need to answer, and don’t lose sight of your objective by getting stuck on making the technology work. Keep your team from becoming too broad or all-encompassing, so you can avoid scope creep and its project failure button: a constantly changing set of requirements flowing from the business to IT. Make sure all stakeholders agree on the objective and keep everyone focused on driving it to completion.

Get endorsement from the top before you start

Once you have identified the business problem you want to solve, the business team must have endorsement from the top down to get access to all the data needed to complete the project successfully. Get company leadership on board with granting the team access to all relevant business data, so they can find the patterns and relationships that will answer the business questions. They must be given access -- controlled, of course -- but with authority.

Make sure your team has the knowledge necessary to execute the project

Ideally, you will have someone on your team who understands machine learning, has the skills and mindset of a data scientist and can actually work with the data to produce the needed business result. If not, you may be able to solve the problem with your existing system. This is a good time to again step back and consider the business questions you need to answer. You may be able to get the answers you need without machine learning or NLP, just by granting access to the right people inside the organization.

Choose a problem that creates value for the business, have the fortitude to stick with it and you’re already on the right path. And remember, a successful project is so regardless of its scope. Don’t set yourself up to fail by biting off a chunk that’s too big. After all, it’s better to succeed in a small project than to fail over and over on a grand scale.