Last week, I wrote a book review on "The Phoenix Project", which looks to DevOps to solve a cross industry problem that a few fashion forward shops — most notably Etsy — have solved. Many more shops do not even recognize it as a problem. The reason this perspective continues to persist, in my opinion, is a combination of fear, misunderstanding and disbelief from IT practitioners and both IT and business leaders.
What is both funny and sad at the same time, is that a solution exists to this problem. It is not a quick fix, but there are very few, if any, quick fixes to real problems. The real problem is this — the lack of an enterprise "continuous deployment" strategy that brings them to the ability to deploy releases, enhancements, patches, fixes and upgrades up to, if not more than, 10 times a day.
Some IT practitioners and leaders do not see deployments as critical bottlenecks. The practitioners look at the individual tasks and say:
"That cannot be automated. There are too many steps and analysis points along the way to automate."
The practitioners cling to their manual tasks like a security blanket. Their denial is a completely false belief that automation tool-sets can't possibly perform either the analysis nor the decision making that they perceive as their primary value proposition. These practitioners fail to recognize two big truths:
- First, the often scary but true reality of career management — The best way to move up the corporate ladder in IT is to work yourself out of a job.
- Second, the presence of a large number of manual and steps and decision trees is a reason to automate, rather than an excuse not to.
"Automating a 5 minute task will not save me any time."
This particular argument shows a lack of understanding the macro-economic equation. The automation process in the way of continuous deployment is neither about automating a single step, nor a series of steps. The point of this form of automation is to eliminate the often unmeasurable space between the tasks. This "dark matter" is present, but undetectable, in almost all IT shops and is the universal force that keeps IT from driving down WIP (work in process).
"That process is so intricate, I don't want to trust that to a computer."
There is no validity to the idea that computers, in the realm of cycling through decision trees, are more susceptible to skipping a step or making an error than a human. The actual misunderstanding that is happening here is that the nay-sayers are thinking of software as if it is hardware. Automation scripts are meant to evolve and grow over time.
Good practitioners of marketing automation and configuration management are happy when a new environmental variant is uncovered. This is the actual point of scripting and automation. Errors only happen once! After a new variant has been identified, the scripts are modified to handle that circumstance, eliminating the same situation from happening in the future.
The perspective of the leaders when looking at the overall system see something slightly different, but still cling to skewed old-school IT precepts.
"We only do deployments once or twice a quarter. Why would you want to automate something that only comes up infrequently?"
This question comes with a hidden assumption that happens to be false — the infrequent nature of deployments are at the behest of the business. The reason why most IT shops are built around the 10 deployments a year (or less), is actually a function of pain avoidance:
- IT pain: Imagine if a traditional IT shop tried to ramp up the frequency of deployments without first driving a culture of continuous deployment. How many 12 hour days, or should I say nights given the typical deployment windows, would the people be willing to sacrifice before they or their spouses would demand they find a new job? The reason these shops do not deploy daily or weekly is the same reason most people don't go to the dentist at that rate — most people hate going to the dentist.
- Business pain: Shops without a culture of continuous associate deployments change with outages. These types of outages are often hard to isolate and eliminate — especially because it often starts the whole deployment process all over again. IT leaders believe that business leaders are upset with the outages, because these tend to go right to the top of the food chain. This is an imprecise judgement. Business leaders would not care even half as much if the MTTR (mean time to resolution) was minutes instead of hours or days. Of course there are cases — credit card authorizations among the most obvious — where a 5 minute outage is highly impactful, but this misses the point. Imagine the difference, if by the time the business VP calls the IT VP to ask about an outage, that the outage has already been repaired.
The idea "It's not a bottleneck, we don't have to do it that often" is backwards. The reason these shops don't deploy that often is because it is a bottleneck. Shops like Etsy and Netflix see their ability to rapidly deploy new feature sets as a competitive advantage because it is one. Can you imagine your business owners turning down the ability to rapidly launch new ideas into the marketplace to prevent start-ups and competitors from waging assaults on your company's competitive position?