silo with "good vibes" painted on it
DataOps evolved from DevOps as a group of methodologies meant to maintain a distributed data architecture. PHOTO: Kyle Glenn

Silos appear on farms, but they also appear in businesses in the form of data silos scattered among different departments. 

Managers now recognize the urgent need to remove data silos, because a holistic understanding of the data flow within a business is crucial for optimizing the customer experience. In the retail sector, a company that is unable to get a complete view of each of its customers because all of its data isn’t readily available in a central venue could face its demise.

Understanding data flow is also critical for regulatory compliance, and that is especially true this year, with the EU’s General Data Protection Regulation (GDPR) due to go into effect in May. If data flows consistently within an organization, it’s easier for managers to determine whether the company is in compliance with regulations.

What Is DataOps?

So what can help managers get a handle on the data silos in their organizations?

One answer may lie in DataOps, a group of methodologies meant to maintain a distributed data architecture. DataOps provide teams with a process framework to manage improvements in data quality and analysis.

DataOps evolved from DevOps, an approach to software engineering designed to bring software developers and operations specialists together in the software development process. I explained how DevOps fits within marketing in an earlier post titled, “How DevOps Streamlines Digital Marketing and Supports Innovation.”

Like DevOps, DataOps uses agile management techniques but with collaborative emphasis on data instead of programming code. Programming languages for predictive analytics, like R and Python, have attracted professionals who are curious about analyzing data sources and about expressing the analysis within programming protocols. Adopting a DataOps approach naturally raises the need to collaborate on making predictive models effective and answering any downstream challenges algorithms can sometimes create.

A Costly Failure to Communicate

One end result of embracing DataOps is a decrease in information silos within an organization. Information silos occur when analysts rely on self-service solutions for analysis but fail to communicate with other departments that may need to use data in similar ways.

A lack of communication also can create costly data errors. One group of users may vet data differently than another group does because they have different criteria as to what constitutes quality data. The cost of errors is high. According to an article in the MIT Sloan Management Review, the cost of bad data can add up to 15 to 25 percent of a company’s revenue.

What marketers must keep in mind is that developer processes like DevOps and DataOps include audits to ensure that content and associated tasks reflect the right level of software requested. The advent of chatbots, apps and mobile devices has made consumer response to media more dynamic, so assessing the programming behind that media avoids misleading conclusions and a tendency to overfocus on metrics that are irrelevant to delivering a good customer experience.

Version Control Enhances Quality

That brings up the importance of version control, a concept that for years developers have embraced with DevOps. Version control platforms like GitHub ensure that teams work on the right version of program code. A recognition of the need for maintaining version control has extended to data science, as practitioners learn to share projects also based on programming languages. That perspective migrates into data usage and how communication about the context of data is repeatedly shared. Thus, version control is a terrific means of ensuring that people start talking about what it takes to deliver programming that supports digital experiences.

Managing data through version control also enhances algorithm security. The ongoing learning capability of an algorithm creates brief time frames for a response to an attack. Knowing what level of data was used in a predictive analytics setting can minimize poor data usage in a model.

The blending of developer protocols will continue. According to Forrester Research’s 2018 predictions for software development, DevOps tools are expected to continue proliferating into devices and various industries. DataOps should follow in step.

Thus, marketers should bolster their knowledge of platforms like GitHub, even if the effort is to just understand how developers share information and to appreciate what programming processes affect production-level code. The end result is being better able to seize opportunities to raise data quality — and business performance — and thereby deliver a meaningful customer experience.