Companies face a common challenge with big data implementations, and it's not the technology. When it comes to implementing these newer big data technologies, the more significant challenge lies with the people and processes. But isn’t that how it’s always been?
Much has been written about how to get the big data technology part right, but the same can't be said for the people side. If you’re just getting started or have already begun your big data journey, here are four characteristics to consider when putting together your big data team.
1. Include Internal and External Data Practitioners
When it comes to staffing your big data team, you don’t have to do it all yourself — nor should you. Why? Because big data technologies and capabilities are still evolving at a rapid pace. It’s next to impossible to stay on top of everything going on in the big data space. It’s more cost-effective and efficient to hire external contractors and service providers whose primary mission is to know big data and its technologies inside and out (so you don’t have to).
So when looking to staff for some of the skillsets and roles discussed below, select a healthy mix of team players from inside and outside your organization, namely:
- Employees: This includes current employees, as well as employees you’re looking to hire
- Contractors/freelancers: A recent report from the Freelancers Union found 53 million people — or 34 percent — in the US workforce are freelancers. They’re predicting that by 2020, up to half of the workforce will be freelancers or contract workers
- Service providers: This could include big data providers (like Cloudera and Hortonworks), cloud providers or any other provider offering X as a service
2. A Healthy Mix of Business and Technical Practitioners
It's easy to think that big data is an IT initiative, but it’s not — and many businesses are learning this the hard way. Stories abound of companies that have jumped on the big data bandwagon, with IT declaring success on one hand, and the business still scratching its head on the other.
There’s no question that implementing big data technologies is fraught with technical challenges, but without the business fully engaged and driving the requirements, your big data initiative is doomed to fail.
A well-balanced big data team will include business and technical practitioners from these six functional areas:
- Platform: Involves configuring and installing the hardware and software infrastructure that will support big data
- Development: Encompasses making the big data environment work. Development activities can include — but aren’t limited to — loading data into Hadoop, customizing open source software projects, writing new code, designing archival strategies within and outside the big data environment, customizing analytics software solutions, performance tuning, and working with the Platform team to help acquire additional functionality
- Data Specialists: The rise of the data specialist has occurred as Platform and Development teams confront the truth that corporate data is more heterogeneous than ever. Rather than straying from their core skill sets to try and understand this data diversity, Platform and Development teams are increasingly turning to data specialists, many of whom have worked on the business side
- Business Stakeholders and Users: Encompasses business people who must find, access, decipher, use, share and deploy data as part of their jobs. Like Data Specialists, they have learned data the hard way, that is, by using it. And as business professionals, they are often consulted to help approve big data decisions, validate business rules, and identify peers who can affirm evolving big data policies, usage requirements and policies
- Governance and Policy: Involves senior leaders and data stewards who advise other functions on how data should or shouldn’t be used, according to larger corporate cultural issues, regulations and fiduciary rules
- Executives: Made up of decision makers who rely on data for strategic and operational decisions, but may or may not be active users
3. Roles Will Change and Evolve as Big Data Industry Changes and Evolves
Many companies have asked whether they need to develop a new, separate big data team, independent of their BI/analytical team(s). The simple answer is no. Big data deserves a seat at the table, but it doesn’t need its own table.
Big data has introduced some new functional roles to our traditional table, such as the big data solution architect, chief data officer, data scientist and Hadoop developer. It’s also requiring new and/or enhanced technical and soft skills. For example:
- Technical skills
- Data science, including mathematics, statistical analysis, statistical programming, analytics modeling techniques, knowledge of data subject matter and the ability to experiment with data without fear
- Data design for handling larger volumes of data
- New software frameworks, such as Hadoop, NoSQL and HBase
- Analytics programming languages
- Soft skills
- Ability to understand business terminology and processes across the company
- Understand corporate strategy and the accompanying KPIs
- Know which business questions are necessary to enable this strategy
- Ability to measure and communicate results
These roles and skills lists are not exhaustive, but they do illustrate how big data is changing company dynamics. Big data technologies are evolving rapidly, and are still a long way from maturity. Until these technologies start to stabilize, we can expect the roles and skills needed to support and effectively use them will continue to evolve.
4. Every Big Data Strategic Initiative Needs One or More Executive Sponsors
The scope of your initiative will inform who the best executive sponsor(s) is for the big data team. If your company is on (or has started down) the big data path, it’s highly likely you have multiple strategic initiatives you’d like to support.
Don’t assume the CIO, CMO or CDO (if you have one) should be the executive sponsor for every big data initiative. And don’t assume a single initiative requires only one sponsor — an initiative could have multiple sponsors. What you can assume, however, is the sponsor will likely change with each initiative, depending on the deliverable.
A Final Note About the Data Scientist
Stop looking for unicorns. For those swept up in the current thinking on the data scientist role — i.e., a quant who can find that insightful needle in your data haystack while delivering a charismatic TED talk — you're more likely to find a mythical creature before you find this data scientist.
Remember: You’re not looking for the data scientist. You’re looking for a collection of talented individuals who can collect data, analyze it, interpret the results and recommend actions. Call them data scientists if you want. These people should be able to support a variety of big data initiatives. One size does not fit all.